Discussion:
Processed: Whether vendor-specific patch series should be permitted in the archive
(too old to reply)
Debian Bug Tracking System
2018-07-23 01:27:06 UTC
Permalink
block 850156 by -1
Bug #850156 [dpkg-dev, debian-policy] Please firmly deprecate vendor-specific series files
850156 was not blocked by any bugs.
850156 was not blocking any bugs.
Added blocking bug(s) of 850156: 904302
--
850156: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850156
904302: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=904302
Debian Bug Tracking System
Contact ***@bugs.debian.org with problems
Bdale Garbee
2018-07-23 17:50:28 UTC
Permalink
The concrete question that I am asking the committee to decide, in my
capacity as a Policy delegate, is whether or not vendor-specific patch
series should be permitted in the Debian archive.
While I am no longer a member of the TC, and my analysis of the
situation consists only of reading this email, I would like to convey a
strongly negative immediate reaction to the idea of allowing
vendor-specific patch series to remain in the Debian archive.

I find Steve's argument that they represent an unhelpful middle ground
particularly persuasive.

Regards,

Bdale
Gunnar Wolf
2018-07-25 03:43:46 UTC
Permalink
As a Debian Policy delegate I hereby request that the Technical
Committee decide whether a proposal that has been submitted to modify
#850156 [n| | ] [dpkg-dev, debian-policy] Please firmly deprecate vendor-specific series files
I am making this request because we have not been able to establish
consensus, but I think this bug is one that we should not just leave
sitting open against debian-policy: if the proposal is eventually
accepted, then leaving the bug open means more vendor-specific series
files in the archive that then have to be removed.
There are currently at least 18 source packages which use
vendor-specific series files. I have not been able to determine an
upper bound.
(...)
Hi,

FWIW, I did a first reading only of the issue, and I find it quite
agreeable. I think it boils down to the principle of least surprise.

There are many alternative and less-obvious ways where a package can
be programmed to act differently depending on where it is being built,
but having it act at source-unpackaging time effectively hides things
and potentially leads to longer head-scratching periods. #ifdef-like
mechanisms are ugly and might carry some reliability issues, but I
think they are preferable as they are so very explicit.

So... I'd like to have some more discussion on this, particularly I'd
like Mike to explain in a nutshell his views. I have not yet read
#850156 (which I really should before stating a definitive viewpoint).
Tollef Fog Heen
2018-07-27 11:32:15 UTC
Permalink
]] Sean Whitton
The concrete question that I am asking the committee to decide, in my
capacity as a Policy delegate, is whether or not vendor-specific patch
series should be permitted in the Debian archive.
There is a broader question of whether source packages should be allowed
to unpack differently on different systems through other means, such as
patch systems implemented in debian/rules (this could be done using
dpkg-vendor(1)). But given that 3.0 (quilt) is now dominant, you might
leave this broader question aside.
I agree with Ian, Steve, and Colin that it is rather surprising for
source packages to unpack differently on a different OS. Source
packages are (at least to most people) a transport format; not entirely
unlike tar, and those do in general not have the property that the
unpacking OS matters. My understanding is also that vendor series are
mostly used by Ubuntu and the experience that Colin and Steve have is
that they are mostly used for the wrong reasons, and cause at least as
many problems as they solve.

I think we should disallow vendor-specific patch series in Debian, and
ask the dpkg developers to kindly deprecate the functionality given the
mentioned concerns.

As for the wider question about patch systems operating through
debian/rules, I'm a lot more comfortable allowing that since you then
run code to get to the patches-applied source code. (I think it's a bad
idea with patches-unapplied packages, but there are use cases for it,
and not all bad ideas should be outlawed.)
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Stuart Prescott
2018-07-27 16:18:51 UTC
Permalink
The concrete question that I am asking the committee to decide, in my
capacity as a Policy delegate, is whether or not vendor-specific patch
series should be permitted in the Debian archive.
I can see that a user might find it surprising that a source package is
unpacked differently on different systems and I agree that it is better that
distro-specific patches bubble up the chain towards upstreams. I agree that
maintainers should avoid setting elephant traps.

It is not, however, within Debian's remit to decide what dpkg in a
derivative distribution does. It is already within the power of the
derivative to accept or reject such patches and they have chosen to accept
these patches. If a derivative does not wish to apply such patches, then
they should simply have dpkg-source not apply them or remove the series file
at package import stage. This is the derivative's choice, not Debian's.

Debian does not micromanage its maintainers and Debian most certainly does
not tell derivatives what to do; however, that is precisely what this
proposal is requesting. Debian should not seek to prevent maintainers doing
something that they have agreed to do in collaboration with downstreams. One
might find vendor.series to be a poor technical solution, but then those who
find it abhorrent might seek to find a better one, rather than attempt to
resort to administrative power.

Essentially, this is a request that the TC overrule both Debian maintainers
*and* derivative maintainers in what they have agreed as a workflow that
obviously works for them. Today, Debian decides to not allow
debian/patches/vendor.series, then tomorrow, a derivative patches dpkg-
source and Debian is asked to decide whether debian/patches/vendor-dammit-i-
want-these-patches-applied.series is allowed in the source package.

The TC has power under §6.1.4 to overrule a developer. It has no power to
overrule a derivative but that is what is really what is being requested.

(Footnote: Sean has referred this under §6.1.3, requesting a decision but
since this is overriding a (set of) maintainer(s) that includes those using
vendor.series and the dpkg maintainers, I assume §6.1.4 applies.)

Stuart
--
Stuart Prescott http://www.nanonanonano.net/ ***@nanonanonano.net
Debian Developer http://www.debian.org/ ***@debian.org
GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Colin Watson
2018-07-27 23:12:08 UTC
Permalink
Post by Stuart Prescott
Debian does not micromanage its maintainers and Debian most certainly does
not tell derivatives what to do; however, that is precisely what this
proposal is requesting.
Is it? What I see is more like a derivative (or at least some of its
developers; Ubuntu has not had a formal discussion about this and I'm
sure we don't have unanimity) asking that Debian update its source
package standards to forbid a feature that was implemented with the best
of intentions to make that derivative's life easier, but which has in
fact turned out to have the reverse effect.

It's technically true that Ubuntu could individually decide to patch out
this feature in both dpkg and all of the source packages that use it,
and that we haven't not done so. That's mainly because it would be a
fair amount of work, not to mention that the patches would be of a
really strange shape (removing ubuntu.series in an Ubuntu patch ...).
Furthermore, since the feature was originally there mainly to improve
collaboration with Ubuntu, or at least that's mostly how it's ended up
being used, if Ubuntu doesn't think it's working well in practice then
Debian processes are *exactly* the correct way to discuss it. The
alternative is a bizarre ping-pong game of patch management behaviour
that would be very confusing for everyone outside the small group of
people up to speed with it, and that would do nobody any good in the
long run.
Post by Stuart Prescott
Debian should not seek to prevent maintainers doing something that
they have agreed to do in collaboration with downstreams.
My memory of the origin of this feature is that the dpkg developer who
originated it asked me if it might help with upstreaming changes from
Ubuntu to Debian, I said something along the lines of "uh, maybe, I
guess, it sounds like that would have some problems?", and then they
went ahead and did it. I don't *think* it was originally a request that
came from derivatives, and I don't remember the sort of collaborative
design that I think you're suggesting here.

I admit that we in Ubuntu didn't push back as hard as in retrospect we
should have done, and some people were positive about the idea; but at
the time, there were a lot of ideas floating around about how to make
the process of upstreaming patches better, and while many of them were
helpful it would have been hard to manage a 100% hit rate. (dpkg-vendor
and related features were implemented around the same time, and I think
those were a good idea and have been helpful.)

That said, it was some time back and I haven't been able to find any
actual transcripts of the conversation, so I'm relying on fallible human
memory and will gladly apologise if I'm wrong. Still, this sort of
design decision should still be open to being revisited when it turns
out to have serious problems in practice.
Post by Stuart Prescott
One might find vendor.series to be a poor technical solution, but then
those who find it abhorrent might seek to find a better one, rather
than attempt to resort to administrative power.
I think Steve and I, at least, have both made it clear what we think
would be better: briefly, apply patches unconditionally but if necessary
give them conditional behaviour that can be controlled at build time.
While this occasionally means making the patch a bit more complicated,
it simplifies management of the patch *series*, and avoids a whole
category of problems that maintainers make in practice. (Certainly I've
been saying essentially this for years when people ask about this kind
of thing on #ubuntu-devel.)

However, features built into the source package format are tempting for
maintainers to use, and so we end up playing whack-a-mole. The only
good fix for that is to remove the feature. It doesn't require a
replacement at the level of the source package format, but it would
require ensuring that packages stop using the feature first, and that's
the kind of interoperability issue that requires a change in policy.
Even if policy doesn't jump straight to forbidding the use of
vendor-specific series, it would make things very much easier if it were
to formally deprecate them.

To me, this is similar to the DEB_AUTO_UPDATE_DEBIAN_CONTROL feature in
CDBS. That was also introduced with good intentions to try to make
maintainers' lives easier, and it was tempting enough that for a while
quite a few people used it; but it proved to have deleterious
side-effects and so was deprecated. (I think this was only ever
deprecated at the level of Lintian and the ftpmaster REJECT-FAQ rather
than actually in policy, but the basic idea seems similar.)
Post by Stuart Prescott
Essentially, this is a request that the TC overrule both Debian maintainers
*and* derivative maintainers in what they have agreed as a workflow that
obviously works for them.
Again, this is a misstatement of what derivative maintainers have in
fact decided, except perhaps by omission.

Some individual package maintainers who care about some derivatives have
made the decision to use this feature. That's not the same thing.
Post by Stuart Prescott
Today, Debian decides to not allow debian/patches/vendor.series, then
tomorrow, a derivative patches dpkg-source and Debian is asked to
decide whether debian/patches/vendor-dammit-i-
want-these-patches-applied.series is allowed in the source package.
A robust decision on this policy question ought to include documenting
the rationale for deprecating the feature, and writing that down in
policy for future travellers so that they can learn from the mistakes of
the past would help to forestall this hypothetical situation. My
opinion from experience with this feature is that those derivative
maintainers would have an easier time if they used patches with
conditional behaviour (or maintained a local branch, of course, although
that was part of what source package management features like this were
trying to reduce).

I'm aware my comments here are mostly Ubuntu-specific; this is not only
because it's what I have most experience of, but also because I'm pretty
sure that ubuntu.series is the only kind of vendor series I've ever seen
in Debian source packages. It's possible I've missed some.
--
Colin Watson [***@debian.org]
Raphael Hertzog
2018-07-29 21:28:37 UTC
Permalink
Hi,
Post by Colin Watson
Post by Stuart Prescott
Debian should not seek to prevent maintainers doing something that
they have agreed to do in collaboration with downstreams.
My memory of the origin of this feature is that the dpkg developer who
originated it asked me if it might help with upstreaming changes from
Ubuntu to Debian, I said something along the lines of "uh, maybe, I
guess, it sounds like that would have some problems?", and then they
went ahead and did it. I don't *think* it was originally a request that
came from derivatives, and I don't remember the sort of collaborative
design that I think you're suggesting here.
As the person who implemented this vendor-specific patch series, I
have to recognize that it was a bad idea. It was easy to implement
and at that time looked like something possibly useful to encourage more
collaboration between Debian an Ubuntu.

But I agree with most of the critics that I have read and I believe
that we should forbid its use in Debian.

Now I'm no longer involved in dpkg, I don't know if Guillem is willing
to deprecate the feature. But at least it should be forbidden at the
policy level.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/
Ian Jackson
2018-08-13 10:16:57 UTC
Permalink
... My opinion from experience with this feature is that those
derivative maintainers would have an easier time if they used
patches with conditional behaviour (or maintained a local branch, of
course, although that was part of what source package management
features like this were trying to reduce).
Obviously I agree with the conclusion, but I would like to propose a
different way of looking at this:

The Debian archive and derivative archives, and the .dsc source
package formats, including "3.0 (quilt)" currently including the
vendor series feature, *are a version control system*. [1]

The vendor series feature *is a way to represent a branch*.
We should consider its merits in those terms, by comparing it to other
technology we have for handling branches.

In the current system [1], the obvious competing way to represent a
branch is for the derivative to actually change the source package.
If vendor series are forbidden, the same effect can always be achieved
that way.

Compared to changing the source package, the vendor series appears to
offer a reduction of maintainer friction. This is a relatively minor
benefit. (Note that resolution of any merge conflicts during rebase,
or whatever, is still necessary; the only benefit is a small degree of
automation and a slight reduction in different files etc. on
maintainer systems.) However, that convenience comes at too high a
price. In particular, the invisibility of the vendor series - that
very lack of friction - means that the vendor series is often wrong.

Conversely, derivatives must already have means for merging or
rebasing their own delta-represented-as-changes-to-the-source-package,
so as to track Debian. (Vendor series cannot, for example, represent
changes to files in debian/.) Those mechanisms are often fairly well
developed, and if they are insufficient they need to be improved [2].
And people can inspect the differences by comparing source packages.

So I think it is nearly always better, even without considering global
effects, to represent any needed derivative delta as a changes source
package, than to represent it as a vendor series.[3]

But forbidding vendor series has the global benefit that no-one who
deals with source packages from Debian derivatives has to write code
to deal with, and spend mental energy on, yet another way to represent
the delta.

Thanks,
Ian.

[1] By modern standards, an appallingly bad one. I would argue that
it all made sense in 1993 (when the best alternative was CVS) but
nowadays we have much much better tools. This is why I am engaged in
a project to provide something much better than .dsc, based on git.

[2] Tracking Debian is IMO still too hard. I am working on that.

[3] That does not mean I think that changing the source package is
always or even mostly the best answer; often other approaches will be
better still. It just means that vendor series are worse than
changing the source package.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Tollef Fog Heen
2018-07-29 05:52:12 UTC
Permalink
]] Stuart Prescott
Post by Stuart Prescott
Essentially, this is a request that the TC overrule both Debian
maintainers *and* derivative maintainers in what they have agreed as a
workflow that obviously works for them. Today, Debian decides to not
allow debian/patches/vendor.series, then tomorrow, a derivative
patches dpkg-source and Debian is asked to decide whether
debian/patches/vendor-dammit-i-want-these-patches-applied.series is
allowed in the source package.
I can see where you are coming from, but I disagree with your
conclusions.

Hypothetically, if a downstream distribution implemented support
vendor.series without there being any support in Debian for it, I don't
think we would disallow vendor.series in Debian. (I think it would be a
bad idea for a downstream to patch such functionality into dpkg, but
there are many bad ideas that should not be forbidden.)
Post by Stuart Prescott
(Footnote: Sean has referred this under §6.1.3, requesting a decision but
since this is overriding a (set of) maintainer(s) that includes those using
vendor.series and the dpkg maintainers, I assume §6.1.4 applies.)
I believe his request might also be considered under §6.1.1, since we're
being asked about a policy change. (After talking to Sean in person, he
said he intended it under §6.1.3, not §6.1.1, though.)
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Sean Whitton
2018-07-30 01:30:04 UTC
Permalink
Hello,
I believe his request might also be considered under §6.1.1, since we're
being asked about a policy change. (After talking to Sean in person, he
said he intended it under §6.1.3, not §6.1.1, though.)
I think technically it's §6.1.3 because according to the policy team
delegation, we decide what goes into the policy manual.

But it certainly falls under at least one of §6.1.1 and §6.1.3, and not
§6.1.4.
--
Sean Whitton
Gunnar Wolf
2018-07-30 03:14:23 UTC
Permalink
Post by Sean Whitton
I believe his request might also be considered under §6.1.1, since we're
being asked about a policy change. (After talking to Sean in person, he
said he intended it under §6.1.3, not §6.1.1, though.)
I think technically it's §6.1.3 because according to the policy team
delegation, we decide what goes into the policy manual.
But it certainly falls under at least one of §6.1.1 and §6.1.3, and not
§6.1.4.
...Although if we make a policy change in this, it will require
cooperation of the dpkg developers if we don't want to go into 6.1.4
(which we don't).

I am explicitly cc:ing Guillem here. Guillem, please take a look at
this bug and give us your PoV!
Sean Whitton
2018-07-30 04:01:35 UTC
Permalink
Hello,
Post by Gunnar Wolf
Post by Sean Whitton
I believe his request might also be considered under §6.1.1, since we're
being asked about a policy change. (After talking to Sean in person, he
said he intended it under §6.1.3, not §6.1.1, though.)
I think technically it's §6.1.3 because according to the policy team
delegation, we decide what goes into the policy manual.
But it certainly falls under at least one of §6.1.1 and §6.1.3, and not
§6.1.4.
...Although if we make a policy change in this, it will require
cooperation of the dpkg developers if we don't want to go into 6.1.4
(which we don't).
Sorry, I don't follow.

Surely policy can disallow things in packages even though those things
have support in dpkg. For example, dpkg lets packages connect to the
Internet to download and embed dependencies, but we forbid that.

Whether or not the feature is removed from dpkg seems orthogonal to the
T.C. bug I filed.
--
Sean Whitton
Gunnar Wolf
2018-07-30 05:14:23 UTC
Permalink
Post by Sean Whitton
Post by Gunnar Wolf
...Although if we make a policy change in this, it will require
cooperation of the dpkg developers if we don't want to go into 6.1.4
(which we don't).
Sorry, I don't follow.
Surely policy can disallow things in packages even though those things
have support in dpkg. For example, dpkg lets packages connect to the
Internet to download and embed dependencies, but we forbid that.
Whether or not the feature is removed from dpkg seems orthogonal to the
T.C. bug I filed.
You are right. Sorry for the noise.

Still, I would like to have Guillem's opinion.
Guillem Jover
2018-07-31 14:44:37 UTC
Permalink
Hi Gunnar,
Post by Gunnar Wolf
Still, I would like to have Guillem's opinion.
Sorry, but I'd rather not participate in the processes I very strongly
disagree with, for a body I don't think should even exist.

Thanks,
Guillem
Stuart Prescott
2018-07-30 04:20:57 UTC
Permalink
Post by Gunnar Wolf
Post by Tollef Fog Heen
I believe his request might also be considered under §6.1.1, since we're
being asked about a policy change. (After talking to Sean in person, he
said he intended it under §6.1.3, not §6.1.1, though.)
I think technically it's §6.1.3 because according to the policy team
delegation, we decide what goes into the policy manual.
But it certainly falls under at least one of §6.1.1 and §6.1.3, and not
§6.1.4.
...Although if we make a policy change in this, it will require
cooperation of the dpkg developers if we don't want to go into 6.1.4
(which we don't).
I am explicitly cc:ing Guillem here. Guillem, please take a look at
this bug and give us your PoV!
It's not just the dpkg maintainers, it is also all the maintainers who are
using this:

$ apt-file search --index-names dsc -x ^/debian/patches/.*\\.series
deluge: /debian/patches/ubuntu.series
fail2ban: /debian/patches/neurodebian-backport.series
filezilla: /debian/patches/debian.series
filezilla: /debian/patches/ubuntu.series
hexchat: /debian/patches/ubuntu.series
libfreenect: /debian/patches/neurodebian-backport.series
libxfce4util: /debian/patches/ubuntu.series
liferea: /debian/patches/ubuntu.series
lilo: /debian/patches/ubuntu.series
mate-power-manager: /debian/patches/ubuntu.series
mate-terminal: /debian/patches/ubuntu.series
mixxx: /debian/patches/ubuntu.series
numix-gtk-theme: /debian/patches/ubuntu.series
packagekit: /debian/patches/ubuntu.series
smuxi: /debian/patches/ubuntu.series
tilix: /debian/patches/ubuntu.series
xchat: /debian/patches/ubuntu.series
xfce4-smartbookmark-plugin: /debian/patches/ubuntu.series

The normal mantra is: "policy is to document standard practice", "policy is
not a stick to beat maintainers with", "policy changes should (normally) not
make packages instabuggy". The current approach is not consistent with
Debian's normal processes for dealing with Policy; vendor.series *is* standard
practice and packages would be made instabuggy for carrying such files if
policy were used as a stick as requested.

(I'm not claiming that this is not a misfeature or that as a project we
shouldn't be helping maintainers and derivatives to get rid of vendor.series,
just that pretending that it's only a policy issue and then using policy as a
back door to overrule maintainers is completely the wrong way of doing it and
undermines both Policy and TC.)

Stuart



dd-list for the above:

Adrien Cunin <***@ubuntu.com>
filezilla

Andrew Starr-Bochicchio <***@debian.org>
deluge (U)

Arne Bernin <***@alamut.de>
libfreenect (U)

Cristian Greco <***@debian.org>
deluge

David Michael Smith <***@gmail.com>
liferea

Debian Desktop Theme Team <numix-gtk-***@packages.debian.org>
numix-gtk-theme

Debian GNOME Maintainers <pkg-gnome-***@lists.alioth.debian.org>
tilix

Debian Multimedia Maintainers <pkg-multimedia-
***@lists.alioth.debian.org>
mixxx

Debian Xfce Maintainers <pkg-xfce-***@lists.alioth.debian.org>
libxfce4util
xfce4-smartbookmark-plugin

Debian+Ubuntu MATE Packaging Team <debian-***@lists.debian.org>
mate-power-manager
mate-terminal
numix-gtk-theme (U)

Emilio Pozuelo Monfort <***@debian.org>
liferea (U)

Evgeni Golov <***@debian.org>
xfce4-smartbookmark-plugin (U)

Free Ekanayaka <***@debian.org>
mixxx (U)

Gianfranco Costamagna <***@debian.org>
xchat

Jeremy Bicha <***@debian.org>
numix-gtk-theme (U)
tilix (U)

Joachim Wiedorn <***@joonet.de>
lilo

John Paul Adrian Glaubitz <***@physik.fu-berlin.de>
mate-power-manager (U)
mate-terminal (U)

Julian Andres Klode <***@debian.org>
packagekit (U)

Lionel Le Folgoc <***@gmail.com>
libxfce4util (U)
xfce4-smartbookmark-plugin (U)

Mark Renouf <***@gmail.com>
libfreenect (U)

Martin Wimpress <***@flexion.org>
mate-terminal (U)

Matteo F. Vescovi <***@debian.org>
mixxx (U)

Matthias Klumpp <***@debian.org>
packagekit
tilix (U)

Mattia Rizzolo <***@debian.org>
hexchat

Mike Gabriel <***@debian.org>
mate-power-manager (U)
mate-terminal (U)
numix-gtk-theme (U)

Mirco Bauer <***@debian.org>
smuxi

Nicolas Bourdaud <***@gmail.com>
libfreenect

Paul Brossier <***@debian.org>
mixxx (U)

Paul Gevers <***@debian.org>
liferea (U)

Petr Baudis <***@ucw.cz>
mate-power-manager (U)

Stefano Karapetsas <***@karapetsas.com>
mate-power-manager (U)
mate-terminal (U)

Vangelis Mouhtsis <***@gnugr.org>
mate-power-manager (U)
mate-terminal (U)

Victor Seva <***@debian.org>
smuxi (U)

Wences Arana <***@debian.org.gt>
mate-terminal (U)

Yaroslav Halchenko <***@onerussian.com>
fail2ban
libfreenect (U)

Yves-Alexis Perez <***@debian.org>
libxfce4util (U)
xfce4-smartbookmark-plugin (U)
--
Stuart Prescott http://www.nanonanonano.net/ ***@nanonanonano.net
Debian Developer http://www.debian.org/ ***@debian.org
GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
Sean Whitton
2018-07-31 00:18:03 UTC
Permalink
Hello,
Post by Stuart Prescott
The normal mantra is: "policy is to document standard practice", "policy is
not a stick to beat maintainers with", "policy changes should (normally) not
make packages instabuggy". The current approach is not consistent with
Debian's normal processes for dealing with Policy; vendor.series *is* standard
practice and packages would be made instabuggy for carrying such files if
policy were used as a stick as requested.
(I'm not claiming that this is not a misfeature or that as a project we
shouldn't be helping maintainers and derivatives to get rid of vendor.series,
just that pretending that it's only a policy issue and then using policy as a
back door to overrule maintainers is completely the wrong way of doing it and
undermines both Policy and TC.)
We're not really in the normal Policy case anymore, because normally
people talk it out and reach consensus. For this bug that simply has
not happened.
--
Sean Whitton
Tollef Fog Heen
2018-07-30 03:51:09 UTC
Permalink
]] Gunnar Wolf
Post by Gunnar Wolf
...Although if we make a policy change in this, it will require
cooperation of the dpkg developers if we don't want to go into 6.1.4
(which we don't).
No, it won't, it's a policy change on what's allowed in the archive, not
what features dpkg is allowed to implement.

I imagine the language we'd want to use is along the lines of «Packages
must not use dpkg's vendor-specific patch series functionality».
Post by Gunnar Wolf
I am explicitly cc:ing Guillem here. Guillem, please take a look at
this bug and give us your PoV!
Thanks!

Cheers,
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Ian Jackson
2018-08-13 10:24:18 UTC
Permalink
Post by Tollef Fog Heen
I believe his request might also be considered under §6.1.1, since we're
being asked about a policy change. (After talking to Sean in person, he
said he intended it under §6.1.3, not §6.1.1, though.)
I think technically it's §6.1.3 because according to the policy team
delegation, we decide what goes into the policy manual.
Insofar as the policy delegation claims to delegate to the policy
editors the final decisions on what goes into policy (rather than
merely the routine task of editing the document) [1], it is ultra
vires. The DPL cannot delegate a power they do not have.

Or to put it another way: even if the policy editors did not refer a
question to the TC, the TC could effectively overrule the policy
editors by using its power in 6.1.1. Obviously this certainly ought
only to be considered after attempts to solve the problem another way
have been fully explored (6.3.6).
But it certainly falls under at least one of §6.1.1 and §6.1.3, and not
§6.1.4.
Obviously I agree with this.

Thanks,
Ian.

[1] I don't read the delegation that way, but for this purpose the
wording of the delegation doesn't matter.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Simon McVittie
2018-08-02 08:46:28 UTC
Permalink
There are currently at least 18 source packages which use
vendor-specific series files. I have not been able to determine an
upper bound.
Here is a survey of packages that do this, based on this search from
Stuart Prescott

$ apt-file search --index-names dsc -x ^/debian/patches/.*\\.series

and my parallel attempt:
https://codesearch.debian.net/search?q=%28diff%7Cpatch%29+path%3Adebian%2Fpatches%2F%28.*%29%5C.series

qjackctl and zlib (in the codesearch) are false positives caused by
older versions existing in the search index.

deluge switches on AppIndicator support by default but only in Ubuntu,
presumably for Unity's benefit. This is in imperative Python code, so it
could easily be runtime-conditional. I suspect this might be better
implemented by detecting a Unity (or AppIndicator-supporting) session
somehow, because people can run non-Unity desktops on Ubuntu, and can
run AppIndicator-supporting desktops on Debian.

fail2ban and libfreenect patch the contents of debian/ (!) to carry
out a semi-automated backport for neurodebian. I think this might be
more appropriately done by merging into a backports git branch, like I
did for flatpak in jessie. Both of these packages also omit the rest
of the patches from neurodebian-backport.series, so either they are
actually using some other mechanism that is not the 3.0 (quilt) vendor
patch series feature, or their backports are (incorrectly?) omitting
the Debian patch series.

filezilla, a file manager, changes how file sizes are displayed to follow
<https://wiki.ubuntu.com/UnitsPolicy> when unpacked on Ubuntu. This
could easily be done with cpp and a compile-time conditional in the
patch if desired.

hexchat, smuxi and xchat, IRC clients, use Ubuntu IRC servers and
channels by default when unpacked on Ubuntu. Again, this could be done
with cpp (for (he)?xchat) or a runtime conditional in C# code (for smuxi)
if desired. smuxi seems to omit one Debian patch when unpacked on
Ubuntu, which is probably a bug (the patch doesn't seem Debian-specific).

libxfce4util uses X-Ubuntu-Gettext-Domain for l10n when unpacked on
Ubuntu. An analogous patch in glib2.0 is applied unconditionally, even
in Debian. I'm not sure which way is better. This is in C code and so
could easily be done with cpp if desired.

liferea, a RSS feed reader, uses Ubuntu RSS feeds by default when unpacked
on Ubuntu. This is a patch to XML configuration files, so would not be
trivial to do any other way while using the same source package for
Ubuntu and Debian (the best idea I have would be to use xsltproc or
xmlstarlet or something, at which point you have two problems).

lilo adjusts user-visible vendor/version strings. This is C code and
could easily be done with cpp, with the bonus that it would work for
any other vendor; I sent a patch to #896081, although it fails to build
(I don't understand why it fails, but I think it might be orthogonal to
my changes).

mate-power-manager hides a preference that, on Ubuntu, is handled by
indicator-power. Like deluge, this seems to be based on an assumption
that Ubuntu is strongly correlated with a desktop environment that uses
particular indicator widgets. This is a patch to XML, so not trivially
replaceable, although you could use xmlstarlet or something.

mate-terminal and tilix, both terminals, have been adapted to Ubuntu
having patched vte to stay with pcre instead of moving to pcre2.
mate-terminal could easily use cpp; tilix is written in D, and I don't
know whether that has a preprocessor. I think both of these would be
better off looking for indications that the vte has been patched in
this way, instead of checking for Ubuntu; otherwise, when Ubuntu
eventually adds pcre2 to main and stops patching vte, they will get
the opposite bug.

mixxx, a DJ-mixing GUI, has a .desktop file that attempts to run under
pasuspender so that it can play audio with direct ALSA, presumably for
smaller/more predictable latency or some other benefit of a simpler audio
path. There is a Debian patch to undo this and just run normally, and an
Ubuntu patch series to undo the Debian patch and use pasupender, with the
comment "Debian-specific as Ubuntu uses PulseAudio by default" in 2010.
The observant will note that the majority of Debian desktops now use
PulseAudio by default, too, so this reasoning might not be valid any more.

numix-gtk-theme uses an Ubuntu-preferred icon theme by patching a
declarative data file, with the comment that Ubuntu developers don't want
this GTK theme to depend on a corresponding icon theme. This looks like
it might indicate a missing (Debian) dependency, tbh... This change is
not in imperative code, but would be easy to make with sed if desired.

packagekit changes various repository and help URLs to be appropriate
for Debian or Ubuntu. This is in declarative data files, so cannot be
done with cpp, although it would probably be straightforward to do with
sed.

xfce4-smartbookmark-plugin changes the bug tracker URL to be appropriate
for Ubuntu. This is in C code and could be done with cpp if desired.

smcv
Wouter Verhelst
2018-08-02 22:17:52 UTC
Permalink
Post by Simon McVittie
mate-terminal and tilix, both terminals, have been adapted to Ubuntu
having patched vte to stay with pcre instead of moving to pcre2.
mate-terminal could easily use cpp; tilix is written in D, and I don't
know whether that has a preprocessor.
It does not, but it does have a mechanism to have a somewhat limited
subset of D code be evaluated at compile time (which is actually far
more powerful than a preprocessor)

It also has an explicit mechanism for conditional compilation, which
would apply here; https://dlang.org/spec/version.html has the details on
how that works.
--
Could you people please use IRC like normal people?!?

-- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
Hacklab
Paul Gevers
2018-08-04 08:54:42 UTC
Permalink
Hi,

[As the main liferea uploader and the one that implemented this there.]
Post by Simon McVittie
liferea, a RSS feed reader, uses Ubuntu RSS feeds by default when unpacked
on Ubuntu. This is a patch to XML configuration files, so would not be
trivial to do any other way while using the same source package for
Ubuntu and Debian (the best idea I have would be to use xsltproc or
xmlstarlet or something, at which point you have two problems).
I don't object an alternative way of doing this, however, I am not going
to do the work. The reason why I did this was to remove the (packaging)
delta between Debian and Ubuntu because the package was often behind in
Ubuntu because of large (semi-) Ubuntu specific changes. Luckily the
technical code has been un-delta-ed some time ago, but if this is
forbidden I'll just drop the patch unless I get a patch that can be
up-streamed for this (I expect upstream to be helpful here).

Paul
Tollef Fog Heen
2018-08-04 08:19:03 UTC
Permalink
Hi,

The Debian Technical Committee has been asked to rule on whether to
disallow the use of dpkg's vendor-specific patch support in the Debian
archive. As you either maintain or are interested in a package that
uses this support, you might want to be aware of this discussion and
possibly participate. The bug number is 904302.

Best regards,
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Tollef Fog Heen
2018-08-09 19:22:15 UTC
Permalink
]] Sean Whitton
The concrete question that I am asking the committee to decide, in my
capacity as a Policy delegate, is whether or not vendor-specific patch
series should be permitted in the Debian archive.
It's now been five days since I mailed the various package
maintainers. I intend to write up a resolution and then call for a vote
in the not-too-distant future, so if there is anything we have not
covered in the discussion so far, please chime in sooner rather than
later
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Margarita Manterola
2018-08-15 09:35:40 UTC
Permalink
Post by Tollef Fog Heen
It's now been five days since I mailed the various package
maintainers. I intend to write up a resolution and then call for a vote
in the not-too-distant future, so if there is anything we have not
covered in the discussion so far, please chime in sooner rather than
later
I don't have much to add, but I want to state my opinion.

I agree with what has been said that the effect of the vendor series is
surprising and causes confusion when people first encounter this
feature.
The reason for this (as has been mentioned) is that it's unexpected that
something would unpack differently in different environments while it's
expected that things will build differently in different environments.

It would be nice if we could carry out a more generic decision based on
this: that source packages should unpack the same regardless of the
derivative on which they are unpacked. However, I don't think we have
the authority for such a ruling, as we can only rule about Debian, not
about derivatives. However, I believe we can still recommend it without
authority.

Apart from this, the concern that has been raised about making packages
instabuggy is valid. I would like our decision to include that this
should
be SHOULD first, giving maintainers a window of time to fix their
packages
and only become a MUST once those packages have been fixed (or something
along those lines, maybe after buster is released).
--
Regards,
Marga
Sean Whitton
2018-08-17 16:03:14 UTC
Permalink
Hello Marga,
Post by Margarita Manterola
Apart from this, the concern that has been raised about making packages
instabuggy is valid. I would like our decision to include that this
should
be SHOULD first, giving maintainers a window of time to fix their
packages
and only become a MUST once those packages have been fixed (or something
along those lines, maybe after buster is released).
I'd like to suggest giving a window of time. Otherwise the policy
process alone won't really have the authority to switch from
should->must.
--
Sean Whitton
Tollef Fog Heen
2018-09-15 17:06:16 UTC
Permalink
]] Tollef Fog Heen
Post by Tollef Fog Heen
]] Sean Whitton
The concrete question that I am asking the committee to decide, in my
capacity as a Policy delegate, is whether or not vendor-specific patch
series should be permitted in the Debian archive.
It's now been five days since I mailed the various package
maintainers. I intend to write up a resolution and then call for a vote
in the not-too-distant future, so if there is anything we have not
covered in the discussion so far, please chime in sooner rather than
later
That turned out to be in the rather more distant future than I
intended. Apologies about that.

A first draft is below, feedback on wording and content appreciated.

=== DRAFT Resolution ===

Vendor-specific patch series are a feature of dpkg that can be used to
apply a different series of quilt patches when the source package is
unpacked on different systems. Since Debian source packages are usually
treated as a pure transport format (like tar), this property can cause
confusion and frustration for users. Examples could be if only the
series file for one vendor is updated, or a source package is unpacked
on one system and then transferred to a system with a different vendor
for debugging.

The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done as part of the build process, using current and future practices
such as patches with conditional behaviour, patching of files during the
build, rather than at source unpacking time.

Since this feature is used by several packages today, we need a
reasonable transition period. They will be considered buggy from when
this resolution is accepted, but it will not be considered severe enough
to warrant immediate removal from Debian. After Buster is released, use
of a vender-specific patch series will be a violation of a MUST
directive in Debian policy.

The Committee therefore resolves that:

1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free).

This should be implemented in Debian Policy by declaring that a a
package SHOULD NOT contain a non-default series file.

2. After Buster is released, use of the vendor-specific patch series
feature is forbidden in the Debian archive.

This should be implemented in Debian Policy by declaring that a a
package MUST NOT contain a non-default series file.

=== End DRAFT Resolution ===
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
David Bremner
2018-09-15 19:14:57 UTC
Permalink
Post by Tollef Fog Heen
The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done as part of the build process, using current and future practices
such as patches with conditional behaviour, patching of files during the
build, rather than at source unpacking time.
Does it need to be said that producing different source packages for
different distributions is also perfectly acceptable? A narrow reading
of this paragraph might give the impression that it is not. One option
would be say something like

... on different distributions, but this difference should not be
done at source unpacking time but e.g. as part of the build
process...

Arguably this concern is moot because everyone (TM) understands that a
package being built refers to going from a source package to one or more
binary packages.
Wouter Verhelst
2018-09-16 11:10:37 UTC
Permalink
Post by David Bremner
Post by Tollef Fog Heen
The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done as part of the build process, using current and future practices
such as patches with conditional behaviour, patching of files during the
build, rather than at source unpacking time.
Does it need to be said that producing different source packages for
different distributions is also perfectly acceptable?
I think that "different distributions" is by definition outside the authority
of the TC, and therefore it shouldn't even be mentioned in the resolution ;-)
--
Could you people please use IRC like normal people?!?

-- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
Hacklab
Sean Whitton
2018-09-16 18:19:27 UTC
Permalink
Hello,
Post by Tollef Fog Heen
A first draft is below, feedback on wording and content appreciated.
LGTM, thank you.
--
Sean Whitton
Philip Hands
2018-09-16 20:00:12 UTC
Permalink
Post by Tollef Fog Heen
This should be implemented in Debian Policy by declaring that a a
^^^
You've this doubled 'a' on two occasions in this text.

Presumaly we would not want to see new packages adopting the use of
vendor-specific patch series prior to Buster.

Do we need to make the "SHOULD NOT" conditional on the package already
having a vendor-specific patch series at the time of this resolution?

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY
Tollef Fog Heen
2018-09-18 20:06:21 UTC
Permalink
]] Philip Hands
Post by Philip Hands
Post by Tollef Fog Heen
This should be implemented in Debian Policy by declaring that a a
^^^
You've this doubled 'a' on two occasions in this text.
I'll fix that, thanks for spotting it.
Post by Philip Hands
Presumaly we would not want to see new packages adopting the use of
vendor-specific patch series prior to Buster.
Do we need to make the "SHOULD NOT" conditional on the package already
having a vendor-specific patch series at the time of this resolution?
I think that just adds needless complexity and assumes that maintainers
will want to add bugs to their package. I really hope that's not the
case, so I don't think it's worthwhile to add extra language for it.
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Philip Hands
2018-09-19 12:39:23 UTC
Permalink
Post by Tollef Fog Heen
]] Philip Hands
Post by Philip Hands
Post by Tollef Fog Heen
This should be implemented in Debian Policy by declaring that a a
^^^
You've this doubled 'a' on two occasions in this text.
I'll fix that, thanks for spotting it.
Post by Philip Hands
Presumaly we would not want to see new packages adopting the use of
vendor-specific patch series prior to Buster.
Do we need to make the "SHOULD NOT" conditional on the package already
having a vendor-specific patch series at the time of this resolution?
I think that just adds needless complexity and assumes that
maintainers will want to add bugs to their package. I really hope
that's not the case, so I don't think it's worthwhile to add extra
language for it.
Well, I was thinking that we could simply state the MUST NOT as the
general case straight away, but with a limited exception for packages
that already contain the bug now, where that exception expires with the
release of Buster.

Your approach seems to require a timely update to policy post-buster,
whereas if there's a conditional in policy there would be no urgency to
removing it, so it can be done at the convenience of the policy
maintainers.

On reflection this seems like we're straying into micro-management.
Should we really be determining the detail of how this is done in
policy?

How about this instead:

The Committee therefore resolves that:

1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free),
however existing use of this feature in packages should not be
considered release critical until after the release of Buster.

Possibly also with something like this?:

Post-Buster this should be implemented in Debian Policy by
declaring that a package MUST NOT contain a non-default series
file.

The approach adopted to allow existing usage is left to the
discretion of the Policy Maintainers.

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY
Ian Jackson
2018-09-19 12:53:37 UTC
Permalink
Post by Philip Hands
Post-Buster this should be implemented in Debian Policy by
declaring that a package MUST NOT contain a non-default series
file.
The approach adopted to allow existing usage is left to the
discretion of the Policy Maintainers.
Nit-picking, you might want to say

The approach adopted to tolerating existing usage before then
is left to the discretion of the Policy Maintainers.

Ian.
Tollef Fog Heen
2018-10-02 17:48:27 UTC
Permalink
]] Philip Hands
Post by Philip Hands
On reflection this seems like we're straying into micro-management.
Should we really be determining the detail of how this is done in
policy?
Given the entire decision has been delegated to us, I think we should,
yes. If we'd just been asked to decide on a matter of technical policy,
that would have been slightly different.
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Sean Whitton
2018-10-02 19:52:43 UTC
Permalink
Hello,
Post by Tollef Fog Heen
Given the entire decision has been delegated to us, I think we should,
yes. If we'd just been asked to decide on a matter of technical policy,
that would have been slightly different.
I think that the wording of my initial mail was ambiguous with regard to
which one of these was being requested.

I would be grateful if you would micromanage just enough that there
isn't anything controversial left for people to disagree about :)
--
Sean Whitton
Ian Jackson
2018-10-03 17:16:08 UTC
Permalink
Post by Sean Whitton
I would be grateful if you would micromanage just enough that there
isn't anything controversial left for people to disagree about :)
That seems like a generally good rule for TC decisions. It avoids
another aspect of the same issue coming back to the TC; and it avoids
setting in stone things that haven't been thought through in the TC
context (and which are best answered by whatever the usual process
is),

In this case I don't think the details (of the transition away from
use of this feature, or the policy wording) are controversial; except
that the timetable, and the consequences at various times for a
package that still has a vendor series, might well be.

No-one seems to have proposed anything but "bug, RC after buster", but
of course opponents of the change are focusing on that and if the TC
just says "these are a bad thing" then an opponent of the change might
well reasonably say "OK I agree this is a bug but it is not RC" or "I
intend to fix this in buster+2".

So, borrowing Phil's text and editing slightly:

1. Presence of any dpkg vendor-specific patch series is a bug for
packages in the Debian archive (including contrib and non-free).
Such a bug should be considered release critical, but not until
after the release of Buster.

The consequences for what to write in policy, want to do in lintian,
how the release team should handle these bugs, etc., all follow
clearly from that text, except for implementaion details that can be
thrashed out in the relevant fora.

I changed "use of [the] feature" to "presence of [the] series" to
avoid the possibility that someone would disingenuously argue that a
series.ubuntu file, in a package in Debian, is not "use" of the
feature.

Ian.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Adrian Bunk
2018-10-03 19:41:11 UTC
Permalink
Post by Tollef Fog Heen
...
1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free),
This misses an important part of the previous proposal:

The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done as part of the build process, using current and future practices
such as patches with conditional behaviour, patching of files during the
build, rather than at source unpacking time.

Otherwise it might sound as if any kind of vendor-specific patching
would be no longer be allowed in the Debian archive.
Post by Tollef Fog Heen
however existing use of this feature in packages should not be
considered release critical until after the release of Buster.
...
We might disagree whether or not the whole change is a good idea, but if
it is done I do not see any good reason for waiting until after the
release of buster.

Policy should in any case provide guidance what replacement for
vendor-specific patching during the build is recommended instead.

With only 18 packages, submitting 18 RC bugs with patches switching them
to the replacement should then be doable immediately.
Post by Tollef Fog Heen
Cheers, Phil.
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Ian Jackson
2018-10-04 12:32:08 UTC
Permalink
Post by Tollef Fog Heen
Post by Tollef Fog Heen
1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free),
I think Phil was just intending to leave the recitals part alone, and
proposing only a change to the operative part - not to delete the
recitals.
Post by Tollef Fog Heen
The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done as part of the build process, using current and future practices
such as patches with conditional behaviour, patching of files during the
build, rather than at source unpacking time.
However, now that we are talking about the recitals I would like to
suggest that the recitals should include *maintaining different source
packages in different distributions* as one of the suggested options.

IMO it is far superior to patches which are conditional (at runtime or
at build-time) on dpkg-vendor and I would not like to see that
perpetuated.

Ian.
Adrian Bunk
2018-10-04 13:49:32 UTC
Permalink
Post by Ian Jackson
Post by Tollef Fog Heen
Post by Tollef Fog Heen
1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free),
I think Phil was just intending to leave the recitals part alone, and
proposing only a change to the operative part - not to delete the
recitals.
Post by Tollef Fog Heen
The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done as part of the build process, using current and future practices
such as patches with conditional behaviour, patching of files during the
build, rather than at source unpacking time.
However, now that we are talking about the recitals I would like to
suggest that the recitals should include *maintaining different source
packages in different distributions* as one of the suggested options.
This is not really applicable here, we are only talking about cases
where Debian maintainer and downstream distribution have in the past
explicitely moved a downstream delta into the package in Debian.
Post by Ian Jackson
IMO it is far superior to patches which are conditional (at runtime or
at build-time) on dpkg-vendor and I would not like to see that
perpetuated.
My understanding of the TC proposal so far is that this would
recommend a 1:1 conversion from vendor-specific patch series
to build-time patching. And as I said, you could even get rid
of the "after buster" part if someone has conversions for all
of the 18 affected packages ready.

My understanding of the TC proposal so far is that this would
recommend a 1:1 conversion from vendor-specific patch series
to build-time patching. And as I said, you could even get rid
of the "after buster" part if someone has conversions for all
of the 18 affected packages ready.

If you want to have the status quo changed, you should IMHO ask
the TC for an explicit decision on that.

The disussion so far has not been about that topic, and adding
a last-minute insertion questioning the status quo into the
decision would not be warranted.

An ambiguous TC decision or policy wording recommending to push such
changes back downstream would also risk conflicts inside Debian if
different people would start arguing with that for whatever side
they personally prefer.
Post by Ian Jackson
Ian.
cu
Adrian

[1] assuming there is a good reason for doing so
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
David Bremner
2018-10-04 14:14:12 UTC
Permalink
Post by Adrian Bunk
My understanding of the TC proposal so far is that this would
recommend a 1:1 conversion from vendor-specific patch series
to build-time patching. And as I said, you could even get rid
of the "after buster" part if someone has conversions for all
of the 18 affected packages ready.
My hope is that the final wording will recommend (demand?) the
conversion from vendor-specific patch series to _something else_,
without taking a position on what that something else should be. The
wording proposed in the last TC meeting (but not yet sent to the bug I
think) did mention build-time patching as an option, but weakened the
initial wording's apparent endorsement of build-time patching as the
recommended method. Before discussing too much further, it might be
helpful to have the revised wording on the bug. If I remember correctly
Tollef was going to post those.

d
Philip Hands
2018-10-04 18:21:07 UTC
Permalink
Post by Ian Jackson
Post by Tollef Fog Heen
Post by Tollef Fog Heen
1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free),
I think Phil was just intending to leave the recitals part alone, and
proposing only a change to the operative part - not to delete the
recitals.
Correct -- sorry if that wasn't clear.
Post by Ian Jackson
Post by Tollef Fog Heen
The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done as part of the build process, using current and future practices
such as patches with conditional behaviour, patching of files during the
build, rather than at source unpacking time.
However, now that we are talking about the recitals I would like to
suggest that the recitals should include *maintaining different source
packages in different distributions* as one of the suggested options.
IMO it is far superior to patches which are conditional (at runtime or
at build-time) on dpkg-vendor and I would not like to see that
perpetuated.
As you say, a separate source package seems like the right way to deal
with this situation.

IMO policy should recomend the use of separate source packages as the
prefered solution to the problem that vendor-specific patch series were
supposed to address.

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY
Ian Jackson
2018-10-04 19:46:16 UTC
Permalink
Post by Philip Hands
IMO policy should recomend the use of separate source packages as the
prefered solution to the problem that vendor-specific patch series were
supposed to address.
That would be great. (I suspect that it is rather controversial,
though, even though I think it is obviously correct...)

If there is consensus in the TC that this is the best approach,
putting something about it in the TC resolution as a recommendation
would have about as much weight as putting it in policy, and avoid
jurisdictional questions.

Ian.
Adrian Bunk
2018-10-04 19:48:06 UTC
Permalink
Post by Philip Hands
...
IMO policy should recomend the use of separate source packages as the
prefered solution to the problem that vendor-specific patch series were
supposed to address.
In this case please make an explicit decision on whether build-time
patching is the recommended replacement for vendor-specific patch
series, or what kinds of build-time patching will no longer be
permitted.

The current situation in the archive is:
- 18 packages with vendor-specific patch series
- an unknown number of packages (e.g. src:gcc-8) that are doing
vendor-specific build-time patching and/or patching based on
other factors like architecture
- > 100 packages that are doing patching and/or configuration
based on dpkg-vendor
- an unknown number of packages (e.g. src:gcc-8) that are doing patching
and/or configuration based on other tools like lsb_release

It is not clear at all which of the above exactly you want to have
removed from the archive and moved as permanent deltas downstream.

The status quo is that everything is permitted,
which is a pretty clear situation.

The TC was asked to make a decision, and a decision turning a clear
situation into a blurry "it is permitted but kinda recommended against"
would only create future conflicts.

A 1:1 vendor-specific patch series -> build-time patching change
would be a mostly technical change. As already said this could
even be implemented before buster.

If the TC wants to additionally change the status quo on the high-level
question whether Debian wants permanent downstream deltas maintained in
Debian or downstream, it should make an explicit decision on that.
Post by Philip Hands
Cheers, Phil.
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Philip Hands
2018-10-05 06:49:47 UTC
Permalink
Post by Adrian Bunk
Post by Philip Hands
...
IMO policy should recomend the use of separate source packages as the
prefered solution to the problem that vendor-specific patch series were
supposed to address.
In this case please make an explicit decision on whether build-time
patching is the recommended replacement for vendor-specific patch
series, or what kinds of build-time patching will no longer be
permitted.
- 18 packages with vendor-specific patch series
- an unknown number of packages (e.g. src:gcc-8) that are doing
vendor-specific build-time patching and/or patching based on
other factors like architecture
- > 100 packages that are doing patching and/or configuration
based on dpkg-vendor
- an unknown number of packages (e.g. src:gcc-8) that are doing patching
and/or configuration based on other tools like lsb_release
It is not clear at all which of the above exactly you want to have
removed from the archive and moved as permanent deltas downstream.
I think it's entirely up to the maintainers of the package (as long as
they avoid vendor-specific patch series in future).

However if someone reads the prohibition against vendor-specific patch
series, and is left wondering what is the best way to deal with this
situation, then it would probably be helpful to give them a hint.

The methods you highlight all seem to suffer from the problem that if a
downstream needs to make a vendor specific change, they need to do an
odd dance where they may have to introduce a delta in order to get the
vendor version out in a timely manner, then they need to get that into
the debian source, and perhaps prompt a no-change release of the Debian
package in order to be able to pick up the change and drop the delta.

It makes much more sense to me to have branches for the debian and
downstream patches side-by-side in one's favourite source control system,
and just build and release whatever one needs, whenever one needs it.

BTW Do we have any way of determining how many packages already deal
with vendor-specific changes in this way?

I'll admit that I've not had to deal with such packages, so feel free to
explain to me why my perception of the situation is utterly deranged.

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY
Ian Jackson
2018-10-05 11:22:53 UTC
Permalink
Post by Adrian Bunk
Post by Philip Hands
IMO policy should recomend the use of separate source packages as the
prefered solution to the problem that vendor-specific patch series were
supposed to address.
In this case please make an explicit decision on whether build-time
patching is the recommended replacement for vendor-specific patch
series, or what kinds of build-time patching will no longer be
permitted.
Surely the TC can recommend X without outlawing Y.
Post by Adrian Bunk
It is not clear at all which of the above exactly you want to have
removed from the archive and moved as permanent deltas downstream.
Speaking personally, I think that packages should almost never look at
dpkg-vendor (or equivalent, eg looking at lsb_release). Any switching
on vendor is probably a bug. The reasons why switching on vendor is
probably wrong have been rehearsed earlier in this discussion.

But I am not asking the TC to outlaw the use of dpkg-vendor et al
because:

* The dpkg vendor series feature is uniquely weird and bad. (Both in
principle, and because of what are arguably lesser design bugs.)

* Unlike the dpkg vendor series feature, the use of dpkg-vendor (or
equivalent) at build- or run-time is not directly in the way of my
project to improve our source code management processes.

* I think an effort to outlaw switching based on vendor would
generate a much bigger fight which is not worth having.

* I hope that my programme of making it easier to handle divergence
properly, by having actually different source code, and carrying
that delta indefinitely, will tempt people away from these kind of
vendor-switching approaches.

Another way to look at this is that in general, where possible, I
think transitions to new things should be done by making the new thing
attractive rather than by going around forbidding or breaking the old
thing.

Sadly the vendor series feature is too broken, and causes too many
problems for others, for that. The problems of switching on vendor
are real but much less severe so taking the carrot rather than stick
approach is much more sensible.

But of course a body like the TC should certainly recommend good
practices rather than troublesome ones.
Post by Adrian Bunk
The TC was asked to make a decision, and a decision turning a clear
situation into a blurry "it is permitted but kinda recommended against"
would only create future conflicts.
We have a lot of things in Debian where one approach is recommended,
but other approaches are permitted or tolerated. This does not in
general seem to result in conflicts.
Post by Adrian Bunk
A 1:1 vendor-specific patch series -> build-time patching change
would be a mostly technical change. As already said this could
even be implemented before buster.
Because I think dpkg-vendor is the wrong answer, I don't want the TC
to recommend that.
Post by Adrian Bunk
However if someone reads the prohibition against vendor-specific patch
series, and is left wondering what is the best way to deal with this
situation, then it would probably be helpful to give them a hint.
Quite.

Ian.
Sam Hartman
2018-10-05 12:40:03 UTC
Permalink
Actually directly switching on vendor seems fairly bad.

However, to the extent that downstream changes can be encapsulated into
options/deltas that someone might want, I think it may often be
reasonable to carry the delta in Debian.

So imagine that Ubuntu and several other downstreams care more about
security and hardening than they do about backward compatibility and
they choose to change a number of gcc and other tool defaults in support
of that. I realize my example is not entirely hypothetical, but I want
to emphasize I have not researched to get all the details right, because
it doesn't matter.

Especially if multiple downstreams or individual users who build from
source might want the change, I think carrying the delta in the Debian
source package can be valuable. It needs to be balanced against a lot
of other concerns.

However, I do tend to agree with Ian that actually turning on that delta
in a specific vendor environment may be best carrief as a
vendor-specific source package.

That said, even there there are tradeoffs.
As an example, Ubuntu tries to use unmodified Debian source packages
where possible. In some cases I think that the maintenance advantages
of doing this and the slight but real political pressure it creates to
push changes upstream to Debian may justify switching on dpkg-vendor.

I think my point here is that there's a lot of complexity, and I'm not
even convinced it would be desirable to recommend against using
mechanisms like dpkg-vendor.
I think what we can hopefully all agree on is that the dpkg vendor patch
series as bad.
Perhaps before we go and recommend something specific we can actually
write up some of these tradeoffs and give people the information they
need to make informed decisions.

And yes, in many cases dgit is the answer. That said, if I were
maintaining the same package both for Debian and for the downstream I
work on, I might well value having a single source tree enough to use
dpkg-vendor or lsb-release or something to switch.
Ian Jackson
2018-10-05 13:45:59 UTC
Permalink
Post by Sam Hartman
So imagine that Ubuntu and several other downstreams care more about
security and hardening than they do about backward compatibility and
they choose to change a number of gcc and other tool defaults in support
of that. I realize my example is not entirely hypothetical, but I want
to emphasize I have not researched to get all the details right, because
it doesn't matter.
Especially if multiple downstreams or individual users who build from
source might want the change, I think carrying the delta in the Debian
source package can be valuable. It needs to be balanced against a lot
of other concerns.
I agree that carrying a switch-on-able delta in the Debian package
would be a good thing there. So, the meat of the change should
definitely go to Debian or maybe even to upstream as a
--with-extra-hardening configure option or some such.

This should be enabled via DEB_BUILD_OPTIONS, or a commented-out line
in the rules file, or by reading /etc/compilers/hardening-enabled at
build- or runtime, or something. Not by looking at `the vendor'.

A scenario why this is needed can be seen from this scenario:

Suppose someone wants to try to maintain a distro which is like
Debian, but which takes some subset of packages from Raspbian.

The obvious way to do this is to import most packages from Debian and
the other packages from Raspbian, and to fix up the bugs which occur
at the interface.

If packages in Debian and Raspbian use dpkg-vendor --is raspbian to
decide whether to do the Raspbian thing, then there is no way to make
this work because there is no correct answer: the answer to whether a
package should do the Raspbian thing depends not only on which distro
we are building or running on, but on which package it is.

In practice if I were maintaining that distro I would be tempted do
some kind of hideous hack to make the output of dpkg-vendor depend on
the name of package we were building. Because how else will I do it ?
Playing whack-a-mole with dpkg-vendor calls would be impractical.

If the Raspbian-specific features are enabled by carrying a changed
source package in Raspbian, then things will mostly DTRT and generally
problems will occur only when the delta-enablement is done via some
kind of indirection, and the indirection messily crosses the `cutoff'
between the Raspbian-originated and Debian-originated packages.

Ie I only have to manually fix the problems that irreducible, not ones
introduced by ill-advised calls to dpkg-vendor.
Post by Sam Hartman
And yes, in many cases dgit is the answer. That said, if I were
maintaining the same package both for Debian and for the downstream I
work on, I might well value having a single source tree enough to use
dpkg-vendor or lsb-release or something to switch.
In that case, that is convenient for you but it is wrong for your
downstreams and users. We should be discouraging such tradeoffs.

Ian.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Wouter Verhelst
2018-10-09 07:44:27 UTC
Permalink
Post by Sam Hartman
That said, even there there are tradeoffs.
As an example, Ubuntu tries to use unmodified Debian source packages
where possible. In some cases I think that the maintenance advantages
of doing this and the slight but real political pressure it creates to
push changes upstream to Debian may justify switching on dpkg-vendor.
I disagree with that, because it forgets why you're pushing things to
Debian.

The point of pushing things upstream is so that you as well as upstream
end up being the same, and the maintenance difference disappears. By
switching on dpkg-vendor, you're *not* the same; instead, you're hiding
your difference. This is not generally helpful; it simply moves the
maintenance burden from Ubuntu to Debian (where it simply does not
belong).

This is papering over a problem rather than fixing one.
Post by Sam Hartman
I think my point here is that there's a lot of complexity, and I'm not
even convinced it would be desirable to recommend against using
mechanisms like dpkg-vendor.
You can easily say "having downstream patches in Debian is generally a
bad idea for the same reasons that having Debian packaging in upstream
sources is generally a bad idea. However, if you must do it, XYZ is the
least terrible way to do so".

[...]
--
Could you people please use IRC like normal people?!?

-- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
Hacklab
Sam Hartman
2018-10-09 14:14:44 UTC
Permalink
That said, even there there are tradeoffs. As an example, Ubuntu
tries to use unmodified Debian source packages where possible.
In some cases I think that the maintenance advantages of doing
this and the slight but real political pressure it creates to
push changes upstream to Debian may justify switching on
dpkg-vendor.
Wouter> I disagree with that, because it forgets why you're pushing
Wouter> things to Debian.

Wouter> The point of pushing things upstream is so that you as well
Wouter> as upstream end up being the same, and the maintenance
Wouter> difference disappears. By switching on dpkg-vendor, you're
Wouter> *not* the same; instead, you're hiding your difference. This
Wouter> is not generally helpful; it simply moves the maintenance
Wouter> burden from Ubuntu to Debian (where it simply does not
Wouter> belong).

I think that we're agreed that evaluating the maintenance burden is
exactly the right criteria.


Imagine a case where the same folks are maintaining a package for
multiple distributions and where the difference is small but important.
In such a case I think our users and the free software community might
best be served by a single repository and switching on something a lot
like dpkg-vendor.

Imagine a case where it's a different set of people doing the work for
Debian than the distribution that wants the change. The Debian
maintainers are not in a good position to test the change and have no
desire to do so. There, switching on vendor seems like the wrong
option.

We're a group of volunteers; we encourage cross-project collaboration
and working together. I believe that the primary consideration should
be what reduces the burden on those doing the work. There are secondary
considerations of course.

--Sam
Wouter Verhelst
2018-10-09 16:16:15 UTC
Permalink
Post by Sam Hartman
That said, even there there are tradeoffs. As an example, Ubuntu
tries to use unmodified Debian source packages where possible.
In some cases I think that the maintenance advantages of doing
this and the slight but real political pressure it creates to
push changes upstream to Debian may justify switching on dpkg-vendor.
Wouter> I disagree with that, because it forgets why you're pushing
Wouter> things to Debian.
Wouter> The point of pushing things upstream is so that you as well
Wouter> as upstream end up being the same, and the maintenance
Wouter> difference disappears. By switching on dpkg-vendor, you're
Wouter> *not* the same; instead, you're hiding your difference. This
Wouter> is not generally helpful; it simply moves the maintenance
Wouter> burden from Ubuntu to Debian (where it simply does not
Wouter> belong).
I think that we're agreed that evaluating the maintenance burden is
exactly the right criteria.
Yes.
Post by Sam Hartman
Imagine a case where the same folks are maintaining a package for
multiple distributions and where the difference is small but important.
In such a case I think our users and the free software community might
best be served by a single repository and switching on something a lot
like dpkg-vendor.
Sure. For one thing though, it's incorrect to assume that the same group
of people will maintain a given package for all eternity, and that the
time when they *do* keep maintaining that package in Debian is the same
as for any number of derivative distributions.

This is all explained in
<https://wiki.debian.org/DebianMentorsFaq#What.27s_wrong_with_upstream_shipping_a_debian.2F_directory.3F>.
While that wiki page talks about the Debian-upstream relationship, many
of the same (kind of) arguments apply equally well to a
downstream-Debian relationship.
Post by Sam Hartman
Imagine a case where it's a different set of people doing the work for
Debian than the distribution that wants the change. The Debian
maintainers are not in a good position to test the change and have no
desire to do so. There, switching on vendor seems like the wrong
option.
We're a group of volunteers; we encourage cross-project collaboration
and working together. I believe that the primary consideration should
be what reduces the burden on those doing the work. There are secondary
considerations of course.
Sure.

Note, I'm not saying that we should outlaw the practice. There are
considerations for doing (or not doing) certain things; an experienced
Debian maintainer can certainly decide that in one particular case,
those considerations do not apply, and then it should be fine to ignore
the advice and do what is right.

But in the general case, I feel that downstream packaging changes belong
downstream, not in Debian; therefore it is best to recommend that, in
the general case, packages in Debian do not switch on dpkg-vendor.
--
Could you people please use IRC like normal people?!?

-- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
Hacklab
Sam Hartman
2018-10-09 17:43:38 UTC
Permalink
Wouter> But in the general case, I feel that downstream packaging
Wouter> changes belong downstream, not in Debian; therefore it is
Wouter> best to recommend that, in the general case, packages in
Wouter> Debian do not switch on dpkg-vendor.

Right. Where as I believe it's not that clear cut and would rather
simply outline a bullet list of tradeoffs for maintainers to consider.

I'd be OK with a general case recommendation though; it's just not how
I'd do it if I were writing text.

And for the record, there are times when I've chosen to ship debian
directories upstream because it was the right thing in that case.
And I've dropped the upstream directory when circumstances changed.
There though, I agree with you that there is a dominant answer: don't
ship debian directories upstream.
Adrian Bunk
2018-10-21 17:48:28 UTC
Permalink
Post by Philip Hands
Post by Adrian Bunk
Post by Philip Hands
...
IMO policy should recomend the use of separate source packages as the
prefered solution to the problem that vendor-specific patch series were
supposed to address.
In this case please make an explicit decision on whether build-time
patching is the recommended replacement for vendor-specific patch
series, or what kinds of build-time patching will no longer be
permitted.
- 18 packages with vendor-specific patch series
- an unknown number of packages (e.g. src:gcc-8) that are doing
vendor-specific build-time patching and/or patching based on
other factors like architecture
- > 100 packages that are doing patching and/or configuration
based on dpkg-vendor
- an unknown number of packages (e.g. src:gcc-8) that are doing patching
and/or configuration based on other tools like lsb_release
It is not clear at all which of the above exactly you want to have
removed from the archive and moved as permanent deltas downstream.
I think it's entirely up to the maintainers of the package (as long as
they avoid vendor-specific patch series in future).
However if someone reads the prohibition against vendor-specific patch
series, and is left wondering what is the best way to deal with this
situation, then it would probably be helpful to give them a hint.
The methods you highlight all seem to suffer from the problem that if a
downstream needs to make a vendor specific change, they need to do an
odd dance where they may have to introduce a delta in order to get the
vendor version out in a timely manner, then they need to get that into
the debian source, and perhaps prompt a no-change release of the Debian
package in order to be able to pick up the change and drop the delta.
This sounds like a very negative description of the standard
Open Source mantra "try to get all your changes upstream".
Post by Philip Hands
It makes much more sense to me to have branches for the debian and
downstream patches side-by-side in one's favourite source control system,
and just build and release whatever one needs, whenever one needs it.
...
Sometimes Debian and downstream maintainer are the same person.

Usually they are not, and then your suggestion won't work.

One common case is Ubuntu requiring different configuration of some
packages, to avoid dependencies of security supported packages in
Ubuntu on packages that are not security supported in Ubuntu.

The fundamental reason for the existence of Raspbian is a vendor
specific difference in all gcc packages. There are also other
packages in Debian that require different configuration for
the lower baseline.

The official source in the archive for a Debian package is the source
tarball, which implies that any general solution used by downstreams
has to be based on the sources in these official tarballs.
Post by Philip Hands
Cheers, Phil.
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Didier 'OdyX' Raboud
2018-10-21 16:04:57 UTC
Permalink
Post by Tollef Fog Heen
1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free),
however existing use of this feature in packages should not be
considered release critical until after the release of Buster.
I don't think this has been mentionned yet:

The TC needs to be careful when outlining what is "release-critical" or not:
AFAIK this is part of what's delegated to the Release Team [0]. So unless the
Release Team defers that specific question to us, I'd recommend to only talk
in terms of Policy vocabulary. That doesn't imply we cannot align our
recommended timelines with Debian suite release dates, of course.

That said, specifically regarding Philip's text, the "should not" is only
emitting a $6.1.5-style opinion towards the RT, which is probably fine.

Cheers,

OdyX

[0] https://lists.debian.org/debian-devel-announce/2014/05/msg00008.html
Tollef Fog Heen
2018-10-23 19:28:40 UTC
Permalink
]] Tollef Fog Heen
Post by Tollef Fog Heen
]] Tollef Fog Heen
That turned out to be in the rather more distant future than I
intended. Apologies about that.
… and again.

Diff from previous one:

- Included separate source packages as an alternative to build time
patching.
- Fixed typo.

I have not included the language about release criticalness of bugs,
due to OdyX' comment about that being the domain of the release team,
not policy.

I hope to call for a vote on this by the end of the week.

Second draft:

=== DRAFT Resolution ===

Vendor-specific patch series are a feature of dpkg that can be used to
apply a different series of quilt patches when the source package is
unpacked on different systems. Since Debian source packages are usually
treated as a pure transport format (like tar), this property can cause
confusion and frustration for users. Examples could be if only the
series file for one vendor is updated, or a source package is unpacked
on one system and then transferred to a system with a different vendor
for debugging.

The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done by using different source packages, or as part of the build
process, using current and future practices such as patches with
conditional behaviour, patching of files during the build, rather than
at source unpacking time.

Since this feature is used by several packages today, we need a
reasonable transition period. They will be considered buggy from when
this resolution is accepted, but it will not be considered severe enough
to warrant immediate removal from Debian. After Buster is released, the
presence of a vender-specific patch series will be a violation of a MUST
directive in Debian policy.

The Committee therefore resolves that:

1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free).

This should be implemented in Debian Policy by declaring that a
package SHOULD NOT contain a non-default series file.

2. After Buster is released, use of the vendor-specific patch series
feature is forbidden in the Debian archive.

This should be implemented in Debian Policy by declaring that a
package MUST NOT contain a non-default series file.

=== End DRAFT Resolution ===
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Didier 'OdyX' Raboud
2018-10-24 06:52:03 UTC
Permalink
Nice, thanks.

A minor nitpicks inline 

Post by Tollef Fog Heen
=== DRAFT Resolution ===
Vendor-specific patch series are a feature of dpkg that can be used to
apply a different series of quilt patches when the source package is
unpacked on different systems. Since Debian source packages are usually
treated as a pure transport format (like tar), this property can cause
confusion and frustration for users. Examples could be if only the
series file for one vendor is updated, or a source package is unpacked
on one system and then transferred to a system with a different vendor
for debugging.
The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done by using different source packages, or as part of the build
process, using current and future practices such as patches with
conditional behaviour, patching of files during the build, rather than
-----------------------^

use "or", as the list of examples has only two items?
Post by Tollef Fog Heen
at source unpacking time.
Since this feature is used by several packages today, we need a
reasonable transition period. They will be considered buggy from when
this resolution is accepted, but it will not be considered severe enough
to warrant immediate removal from Debian. After Buster is released, the
presence of a vender-specific patch series will be a violation of a MUST
--------------------^

vendor
Post by Tollef Fog Heen
directive in Debian policy.
1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free).
(
)
2. After Buster is released, use of the vendor-specific patch series
feature is forbidden in the Debian archive.
(
)
I'm not sure how these are to be interpreted with regards to "the whole Debian
Post by Tollef Fog Heen
2. After Buster is released, packages must not use vendor-specific patch
series.

 but it resonates with the second paragraphs. I can live with the existing
phrasing.

Cheers,
OdyX
Ian Jackson
2018-10-24 12:52:42 UTC
Permalink
...
Post by Tollef Fog Heen
The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done by using different source packages, or as part of the build
process, using current and future practices such as patches with
conditional behaviour, patching of files during the build, rather than
at source unpacking time.
This is all good but can I suggest using the phrase `differing source
packages' rather than `different' ? `Different' might mean source
packages with different source package name. `Differing' more clearly
refers to source packages of the same name but which are different to
each other.

Thanks,
Ian.
Tollef Fog Heen
2018-10-30 20:49:40 UTC
Permalink
Last draft, incorporating the suggestions from Ian and Didier.

=== DRAFT Resolution ===

Vendor-specific patch series are a feature of dpkg that can be used to
apply a different series of quilt patches when the source package is
unpacked on different systems. Since Debian source packages are usually
treated as a pure transport format (like tar), this property can cause
confusion and frustration for users. Examples could be if only the
series file for one vendor is updated, or a source package is unpacked
on one system and then transferred to a system with a different vendor
for debugging.

The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done by using differing source packages, or as part of the build
process using current and future practices such as patches with
conditional behaviour or patching of files during the build rather than
at source unpacking time.

Since this feature is used by several packages today, we need a
reasonable transition period. They will be considered buggy from when
this resolution is accepted, but it will not be considered severe enough
to warrant immediate removal from Debian. After Buster is released, the
presence of a vendor-specific patch series will be a violation of a MUST
directive in Debian policy.

The Committee therefore resolves that:

1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free).

This should be implemented in Debian Policy by declaring that a
package SHOULD NOT contain a non-default series file.

2. After Buster is released, use of the vendor-specific patch series
feature is forbidden in the Debian archive.

This should be implemented in Debian Policy by declaring that a
package MUST NOT contain a non-default series file.

=== End DRAFT Resolution ===
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Anonymous
2018-09-29 15:06:58 UTC
Permalink
Dear Chair

reasoning of this policy is really absurd. The opposite is actually
true. Usage of vendor patches should be encouraged downstream. That's a
free software issue! The goal is to facilitate patches.

If Debian want patches it has to support this process with tools. The
attitude Debian owns all source packages is wrong. Sharing source
packages among different vendors is more efficient. Different patch
series may be the best solution in some cases. This policy decision only
breaks the workflow. Derivatives have to duplicate the whole source
tree. It is a huge burden and waste of resources.

Patch series are supported by git-am and git-format-patch. There is no
better approach to incorporate patches. I fear circumventing the policy
with "QUILT_SERIES=debian/patches/$(dpkg-vendor --query vendor).patch
quilt push -a" in debian/rules. The patch series separates vendor
specific code properly. If policy is against vendor specific code it has
to accept patch series at least. They are a last resort to make
independent patches.

Builds for different vendors are not a standard use case at all. Identic
source after unpacking is possible with dpkg-source --skip-patches
anyways. A hint about different series during unpacking can be useful
but changing policy because someone was confused is unbelievable. Usage
of the right tools is good practice and should not forced with power.

The decision is based on wrong assumptions and implications, arguments
are weak, valid objections ignored. This is abusing Debian policy and
technical committee against free software! Debian needs patches
regardless of policy.

Yours truly
Gunnar Wolf
2018-09-29 22:23:32 UTC
Permalink
Post by Anonymous
Dear Chair
Dear Anonymous,

Although it is of course completely fine for you to contact us
anonymously, in cases such as this one, having a "name" will help your
case. Do you actually use this? Have you worked with the issue? Is it
bothering you?

Anonymous opinions are acceptable. But Debian is a socially cohesive
group of people. It helps us to match opinions with people. Would help
your point.

Anyway, thanks for your mail.
Post by Anonymous
(...)
Patch series are supported by git-am and git-format-patch. There is no
better approach to incorporate patches. I fear circumventing the policy
with "QUILT_SERIES=debian/patches/$(dpkg-vendor --query vendor).patch
quilt push -a" in debian/rules. The patch series separates vendor
specific code properly. If policy is against vendor specific code it has
to accept patch series at least. They are a last resort to make
independent patches.
Well, IMO this would be precisely the _right_ way to do this: The
source you have on disk at source package unpacking time is the same
everywhere, and you can see precisely what would happen when building
in Mint, Ubuntu, Debian or $whatever. This would not be circumventing
policy — It would be following it with minimal friction to what you
already have.
Post by Anonymous
Builds for different vendors are not a standard use case at all. Identic
source after unpacking is possible with dpkg-source --skip-patches
anyways. A hint about different series during unpacking can be useful
but changing policy because someone was confused is unbelievable. Usage
of the right tools is good practice and should not forced with power.
The decision is based on wrong assumptions and implications, arguments
are weak, valid objections ignored. This is abusing Debian policy and
technical committee against free software! Debian needs patches
regardless of policy.
I do not share that feeling; I think we argued constructively with
people that were against this outcome, and while there is not
universal consensus, expressed issues were taken into account.
Ian Jackson
2018-10-02 14:07:19 UTC
Permalink
Post by Anonymous
If Debian want patches it has to support this process with tools. The
attitude Debian owns all source packages is wrong. Sharing source
packages among different vendors is more efficient. Different patch
series may be the best solution in some cases.
I do agree with the underlying ideology behind these ideas. I think
code sharing between different distros in the Debianish ecosystem is
very important and I certainly don't think that `Debian owns all
source packages', whatever that means.

Indeed, in my technical Debian work I am writing tools which I hope
will support people who want to diverge from Debian, and retain and
carry those divergences in the long term. I have long been frustrated
that it is too difficult to do this.

The problem I have with the vendor series feature is narrower and more
technical. For all the reasons I and others have explained, I think
the vendor series feature is a very poor way to support divergence and
diversity. It does more harm than good.

The background to this is that I think that Debian source packages,
which I designed in the mid 1990s and which were since extended with
`3.0 (quilt)' [1], are a clumsy system which has been obsoleted by the
new generation of distributed version control systems, especially git.

.dsc format source packages are bad enough for the newcomer to Debian,
without the very weird vendor patch series feature. And the vendor
patch series feature makes migration to better source code management
tools harder.

So in summary I think the real way to promote divergence by Debian's
derivatives, and code sharing amongst derivatives, is to use to the
full the features of very capable modern revision control systems.

TBH I don't expect this to convince you. And I found many of your
comments rather overblown. It would be helpful if you could avoid
wild accusations.

But, if you really want to help promote software freedom for Debian's
derivatives and users, by addressing issues to do with package source
code management tooling: please consider trying out dgit and maybe
suggesting it to Debian's downstreams as a way to get the source code
from Debian.

Please also consider advocating that Debian maintainers should use
dgit for their uploads. If you're very keen you could come and help
out with work on making git-debrebase become a useful tool for
downstreams.

Ian.

[1] To be clear, although I have a lot of criticisms of `3.0 (quilt)',
it is much better than what was being widely done in Debian
beforehand.
Adrian Bunk
2018-08-16 21:01:56 UTC
Permalink
Hi,

looking at something where I worked on the upstream implementation ages ago:
https://sources.debian.org/src/liferea/1.12.4-1/debian/patches/ubuntu-example-feeds.patch/

It is a common problem that users should be able to get started quickly
after installing a program.

When liferea is started by a user for the first time, the default feedlist
in the locale of the user gets installed as feedlist for the user.

It is clear why a derivative, especially a brand-aware one like Ubuntu,
wants to change this feedlist.

And it is also clear why this change cannot be applied in Debian.

One obvious solution if vendor-specific series files get outlawed in
Debian would be to switch from ubuntu.series to manual patching in
debian/rules based on dpkg-vendor(1).

There are two problems:

First, it replaces a working standard dpkg feature with an own
package-specific implementation.
There are surprisingly many things a maintainer can get wrong,
like some other packages doing debian/rules patching where this
breaks building twice in a row.

Second, it hides the whole situation/problem.
Finding all packages that use vendor-specific patch series is simple
and has been done in this discussion here.
This makes it also simple to find bogus cases where the change should
actually be applied in Debian (or upstream).
How do you find all 3.0 (quilt) packages that have conditional patching
implemented in debian/rules?
There is a broader question of whether source packages should be allowed
to unpack differently on different systems through other means, such as
patch systems implemented in debian/rules (this could be done using
dpkg-vendor(1)). But given that 3.0 (quilt) is now dominant, you might
leave this broader question aside.
I could name a 3.0 (quilt) package where Sean Whitton has done a
maintainer upload this year that has a patch applied conditionally
in debian/rules with a not completely correct patching mechanism.
The name of the package has not been mentioned in this bug so far.

On the more general point, there is no actual rationale given why the
TC should discuss outlawing vendor-specific patch series without also
discussing manual debian/rules patching in 3.0 (quilt) packages based
on dpkg-vendor(1) or other mechanisms.


The whole underlying notion that there would be one source tree that
gets built is also flawed. This is not true in all cases, and can
never be.
For example, someone might want to use a Debian system to investigate a
bug on an Ubuntu system. They might begin by downloading some source
packages from the Ubuntu mirrors. Since they obtained them from Ubuntu,
they will form the reasonable expectation that unpacking these source
packages will get them the code running on the Ubuntu system they are
debugging.
This is not a "reasonable expectation", this is a bogus assumption.
And users should be clearly told that investigating Ubuntu problems
on a Debian system is not a good idea - the supported way for that
is a chroot (or some more sophisticated virtualization solution).

I do not see how this could ever work for a package like src:gcc-8,
that is based on several upstream tarballs and then patches and
configures the code differently based on:
- distribution
- codename of the distribution
- architecture


It might be too complex and not worth the effort, but if anything should
be changed it should actually be changed in the opposite direction:

Replace buggy patching systems implemented in individual packages
with a standard non-buggy implementation in dpkg that supports
more ways of conditional patching.


cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Tollef Fog Heen
2018-08-17 07:49:00 UTC
Permalink
]] Adrian Bunk
Post by Adrian Bunk
Hi,
https://sources.debian.org/src/liferea/1.12.4-1/debian/patches/ubuntu-example-feeds.patch/
It is a common problem that users should be able to get started quickly
after installing a program.
When liferea is started by a user for the first time, the default feedlist
in the locale of the user gets installed as feedlist for the user.
It is clear why a derivative, especially a brand-aware one like Ubuntu,
wants to change this feedlist.
And it is also clear why this change cannot be applied in Debian.
One obvious solution if vendor-specific series files get outlawed in
Debian would be to switch from ubuntu.series to manual patching in
debian/rules based on dpkg-vendor(1).
Or it would mean that Ubuntu would carry a XubuntuY version and do
manual (or automatic, based on whatever tooling they have available)
merges from Debian, marking it clearly as a different work.

[...]
Post by Adrian Bunk
The whole underlying notion that there would be one source tree that
gets built is also flawed. This is not true in all cases, and can
never be.
We are not talking about what's built. We're talking about what's
unpacked. It's well understood that what's unpacked is not always
what's built; build processes can (and do) all kinds of transformations
and is understood to be executable code.
--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Adrian Bunk
2018-08-17 09:03:41 UTC
Permalink
Post by Tollef Fog Heen
]] Adrian Bunk
Post by Adrian Bunk
Hi,
https://sources.debian.org/src/liferea/1.12.4-1/debian/patches/ubuntu-example-feeds.patch/
It is a common problem that users should be able to get started quickly
after installing a program.
When liferea is started by a user for the first time, the default feedlist
in the locale of the user gets installed as feedlist for the user.
It is clear why a derivative, especially a brand-aware one like Ubuntu,
wants to change this feedlist.
And it is also clear why this change cannot be applied in Debian.
One obvious solution if vendor-specific series files get outlawed in
Debian would be to switch from ubuntu.series to manual patching in
debian/rules based on dpkg-vendor(1).
Or it would mean that Ubuntu would carry a XubuntuY version and do
manual (or automatic, based on whatever tooling they have available)
merges from Debian, marking it clearly as a different work.
On the more general point, there is no actual rationale given why the
TC should discuss outlawing vendor-specific patch series without also
discussing manual debian/rules patching in 3.0 (quilt) packages based
on dpkg-vendor(1) or other mechanisms.
If vendor-specific patching (and configuration?) should move from Debian
to deltas carried permanently by derivatives, then this should be
discussed and decided for the general case.

Outlawing a non-buggy patching implementation will in practice only
result in the same patching being done with per-package patching
implementations. These are more likely to be buggy, and make it
impossible to easily search for all packages doing it.
Post by Tollef Fog Heen
[...]
Post by Adrian Bunk
The whole underlying notion that there would be one source tree that
gets built is also flawed. This is not true in all cases, and can
never be.
We are not talking about what's built. We're talking about what's
unpacked. It's well understood that what's unpacked is not always
what's built; build processes can (and do) all kinds of transformations
and is understood to be executable code.
Is it really well understood?

A large part of the email that opened this bug only makes sense under
the assumption that a TC decision will result in what is being unpacked
becoming what will be built.

More exactly, the part from "And secondly" up to and including the
section discussing "#ifdef ubuntu".

For arguments like "will get them the code running on the Ubuntu system"
or "#ifdefs are visible in the actual source code" it doesn't matter
whether the different patching based on the distribution will be done
by dpkg or debian/rules, and the same applies to debian/rules patching
based on the release or the architecture.


cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Iain Lane
2018-08-17 09:30:15 UTC
Permalink
Post by Tollef Fog Heen
Post by Adrian Bunk
One obvious solution if vendor-specific series files get outlawed in
Debian would be to switch from ubuntu.series to manual patching in
debian/rules based on dpkg-vendor(1).
Or it would mean that Ubuntu would carry a XubuntuY version and do
manual (or automatic, based on whatever tooling they have available)
merges from Debian, marking it clearly as a different work.
I would like you to consider - and I think this is part of what Adrian
is raising too¹ - this kind of case where the Debian maintainer *wants*
to support particular derivatives in their source package in Debian and
is willing to test it properly.

Having this facility avoids the need for any kind of source package
delta resolution process needing to take place², which might add
arbitrary delays between a new package being uploaded to unstable and
becoming available in the derivative's unstable suite. It means that the
Debian uploader does not need to become - or to find - a derivative
uploader to perform this resolution. And it avoids maintainers having to
cook up their own solution if they want to do it at build time without
tool support.

FAOD I am not challenging that the drawbacks of vendor-specific patch
series as outlined in this bug exist - just saying that Adrian's point
has some merit in that having this *kind* of support (perhaps not this
specific implementation) easily available is useful.

TBH I'm not sure what I'd ask -ctte to do with this argument. If you do
decide to outlaw the vendor-specific series, maybe advice (6.1.5) to
relevant developers to consider designing a way to better support
derivative specific patches within Debian.

Cheers for handling this issue,
--
Iain Lane [ ***@orangesquash.org.uk ]
Debian Developer [ ***@debian.org ]
Ubuntu Developer [ ***@ubuntu.com ]

¹ sorry for putting words in your mouth if this isn't what you meant :-)
² in Ubuntu's case this is not automatic; a human uploader needs to at
the very least review the output of merge-o-matic).
Ian Jackson
2018-08-17 11:32:06 UTC
Permalink
Post by Iain Lane
Post by Tollef Fog Heen
Post by Adrian Bunk
One obvious solution if vendor-specific series files get outlawed in
Debian would be to switch from ubuntu.series to manual patching in
debian/rules based on dpkg-vendor(1).
Or it would mean that Ubuntu would carry a XubuntuY version and do
manual (or automatic, based on whatever tooling they have available)
merges from Debian, marking it clearly as a different work.
Quite.

If an Ubuntu user downloads the liferea source code from Debian, to
see what Debian's version is like, they should *not* get the Ubuntu
branding and default feed changes.

What this example shows is that, when a derivative wants a program to
behave differently for these kind of reasons, this should generally be
done by *actually changing the source code*.

Ubuntu already has machinery for automatically merging from Debian.
That machinery should be used. And improved, if necessary.
Post by Iain Lane
I would like you to consider - and I think this is part of what Adrian
is raising too¹ - this kind of case where the Debian maintainer *wants*
to support particular derivatives in their source package in Debian and
is willing to test it properly.
The matter might be different for changes which are to deal with the
presence or absence of particular things in the derivative. In that
case, the change can simply be made to Debian. But it should be
conditional not on dpkg-vendor, but on whatever the actual difference
is.

In general, I think package builds should not pay any attention to
dpkg-vendor. Can I please add that request to the TC's deliberations ?
Post by Iain Lane
Having this facility avoids the need for any kind of source package
delta resolution process needing to take place
This is, of course, false.

The vendor series file *is a source package delta resolution process*.
It's just a terrible one.
Post by Iain Lane
which might add arbitrary delays between a new package being
uploaded to unstable and becoming available in the derivative's
unstable suite.
If new updates are not be merged automatically, by a robot, in a
timely way, then that is a tooling problem.

Note that if there is a vendor series, *other* changes made in Debian
to the upstream files in the package will be silently ignored.
This is incredibly counterintuitive and undesriable.
Post by Iain Lane
TBH I'm not sure what I'd ask -ctte to do with this argument. If you do
decide to outlaw the vendor-specific series, maybe advice (6.1.5) to
relevant developers to consider designing a way to better support
derivative specific patches within Debian.
"Derivative-specific patches" covers two kinds of changes:

1. Changes which are direct differences of policy or preference in the
derivative, such as changes to the default liferea feeds, or changes
to which package(s) are available.

2. Changes which are consequential on other changes. Eg, changes to
cope with different (build-)dependencies.

No changes of type (1) should never appear in Debian source packages,
for the reasons I have explained. Therefore the mechanism you are
suggesting "relevant developers" should design ought not to exist.
Instead, if carrying patches downstream is too hard, that should be
fixed. And indeed I and others are working on that.

Changes of type (2) should be ideally be dealt with by detecting the
situation at build-time where possible, but that should be done by
looking at which of the alternative build dependencies is installer,
or whatever, not by checking dpkg-vendor.

Ian.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Sean Whitton
2018-08-17 16:04:47 UTC
Permalink
Hello,
Post by Ian Jackson
In general, I think package builds should not pay any attention to
dpkg-vendor. Can I please add that request to the TC's deliberations ?
This is about package builds, not the unpacking of source packages, so
is surely a completely separate issue (which we should discuss at the
level of Debian Policy, not refer directly to the T.C.).
--
Sean Whitton
Sean Whitton
2018-08-17 15:58:49 UTC
Permalink
Hello,
Post by Adrian Bunk
For example, someone might want to use a Debian system to investigate a
bug on an Ubuntu system. They might begin by downloading some source
packages from the Ubuntu mirrors. Since they obtained them from Ubuntu,
they will form the reasonable expectation that unpacking these source
packages will get them the code running on the Ubuntu system they are
debugging.
This is not a "reasonable expectation", this is a bogus assumption.
And users should be clearly told that investigating Ubuntu problems
on a Debian system is not a good idea - the supported way for that
is a chroot (or some more sophisticated virtualization solution).
People don't expect that running dpkg-buildpackage on a Debian system
would get them the binary package running on an Ubuntu system. That's
different from the expectation that the source they get is the source
running on their Ubuntu system.
--
Sean Whitton
Adrian Bunk
2018-08-17 16:36:11 UTC
Permalink
Post by Sean Whitton
Hello,
Post by Adrian Bunk
For example, someone might want to use a Debian system to investigate a
bug on an Ubuntu system. They might begin by downloading some source
packages from the Ubuntu mirrors. Since they obtained them from Ubuntu,
they will form the reasonable expectation that unpacking these source
packages will get them the code running on the Ubuntu system they are
debugging.
This is not a "reasonable expectation", this is a bogus assumption.
And users should be clearly told that investigating Ubuntu problems
on a Debian system is not a good idea - the supported way for that
is a chroot (or some more sophisticated virtualization solution).
People don't expect that running dpkg-buildpackage on a Debian system
would get them the binary package running on an Ubuntu system. That's
different from the expectation that the source they get is the source
running on their Ubuntu system.
The main misconception is that there would always be *the* source.

Steps you might have before the compilation starts:
1. dpkg unpacks upstream sources
2. dpkg applies patches
3. debian/rules unpacks upstream tarballs as part of the build
4. debian/rules applies patches based on distribution
5. debian/rules applies patches based on release
6. debian/rules applies patches based on architecture

What is "the source running on their Ubuntu system" for src:gcc-8?

This package skips steps 1 and 2, but does all of steps 3-6.

After step 2 you have 3 upstream tarballs plus a debian/ with a
pretty sophisticated custom machinery - that's not useful for
seeing what gets built.

If you want to look after step 6 at the source "running on your system",
what you will see depends on your:
- distribution: Debian or Ubuntu?
- release: stretch or buster/sid?
- architecture: amd64 or arm64?
Post by Sean Whitton
Sean Whitton
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Ian Jackson
2018-08-17 18:33:02 UTC
Permalink
Post by Adrian Bunk
The main misconception is that there would always be *the* source.
1. dpkg unpacks upstream sources
2. dpkg applies patches
3. debian/rules unpacks upstream tarballs as part of the build
4. debian/rules applies patches based on distribution
5. debian/rules applies patches based on release
6. debian/rules applies patches based on architecture
I disagree that (4) should ever be relevant. There should never be
any patchese applied conditionally based on dpkg-vendor, for the
reasons I explained very recently in response to the liferea example./

We don't ever do (5), do we ? Please tell me we don't. We can have
different source code in our different releases.

I can see that (6) might be needed in some exceptional cases but
normally there is a better way.

Ian.
--
Ian Jackson <***@chiark.greenend.org.uk> These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.
Adrian Bunk
2018-08-17 20:46:23 UTC
Permalink
Post by Ian Jackson
Post by Adrian Bunk
The main misconception is that there would always be *the* source.
1. dpkg unpacks upstream sources
2. dpkg applies patches
3. debian/rules unpacks upstream tarballs as part of the build
4. debian/rules applies patches based on distribution
5. debian/rules applies patches based on release
6. debian/rules applies patches based on architecture
I disagree that (4) should ever be relevant. There should never be
any patchese applied conditionally based on dpkg-vendor, for the
reasons I explained very recently in response to the liferea example./
We don't ever do (5), do we ? Please tell me we don't. We can have
different source code in our different releases.
For packages like src:firefox-esr the same source code might
be maintained to support releases ranging from oldoldstable to
unstable - 52.8.0esr-1~deb7u1 is a security update for wheezy
that is technically a backport of a package in buster/sid.

I do not know whether firefox-esr does patching based on release right
now, but this is a case where one package is being used to provide
security support for up to 4 different Debian releases at the same time.

I would also not be surprised if some package would do different
patching based on release for easy rebuilding in stable-backports,
this would sound like a natural solution to me.
Post by Ian Jackson
I can see that (6) might be needed in some exceptional cases but
normally there is a better way.
As I said, src:gcc-8 does all of steps 3-6.

Much of the relevant code is in
https://sources.debian.org/src/gcc-8/8.2.0-4/debian/rules.patch/

Things are even more interesting due to this being a debian/ shared
between Debian and Ubuntu which are using different upstream sources.[1]
Post by Ian Jackson
Ian.
cu
Adrian

[1] GFDL
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Sean Whitton
2018-08-18 18:31:06 UTC
Permalink
Hello,
Post by Adrian Bunk
The main misconception is that there would always be *the* source.
1. dpkg unpacks upstream sources
2. dpkg applies patches
3. debian/rules unpacks upstream tarballs as part of the build
4. debian/rules applies patches based on distribution
5. debian/rules applies patches based on release
6. debian/rules applies patches based on architecture
What is "the source running on their Ubuntu system" for src:gcc-8?
This package skips steps 1 and 2, but does all of steps 3-6.
But all of steps 3--6 are part of the package build.

"The source" is what you get after steps 1 and 2.
--
Sean Whitton
Sean Whitton
2018-08-19 19:11:01 UTC
Permalink
Hello Adrian,
Why is "The source" what you get after dpkg applied patches,
but before debian/rules applied patches?
Because we define the package build as something that starts when you
invoke the debian/rules script.
For a user it doesn't make a difference which tool applies the patches.
In my mind, it does; it matters whether or not it is part of the package
build. That's just my expectation of what reasonable users will think.

We're discussing what users will reasonably expect. If you and I have
different intuitions about the expectations that reasonable users will
form, we're going to have to agree to disagree. Neither of us is in a
position to conduct field research on this question (afaik!).
Note that you were also arguing based on a different source
For example, someone might want to use a Debian system to investigate a
bug on an Ubuntu system. They might begin by downloading some source
packages from the Ubuntu mirrors. Since they obtained them from Ubuntu,
they will form the reasonable expectation that unpacking these source
packages will get them the code running on the Ubuntu system they are
debugging.
This would be useful for debugging problems.
But it is important to understand that in the general case there will
always be cases where the code running on your system will depend on
the architecture of your system - after applying patches the sources
might be architecture-specific.
Unless I'm missing something, that can only be true when the application
of patches to which you refer occurs during the package build.
--
Sean Whitton
Adrian Bunk
2018-08-19 20:15:56 UTC
Permalink
Post by Sean Whitton
...
...
For a user it doesn't make a difference which tool applies the patches.
In my mind, it does; it matters whether or not it is part of the package
build. That's just my expectation of what reasonable users will think.
We're discussing what users will reasonably expect. If you and I have
different intuitions about the expectations that reasonable users will
form, we're going to have to agree to disagree.
...
The user sees that the sources get unpacked, and that patches get applied.

You are saying that the reasonable user will expect that these patched
sources might not be the sources that will get built, and that a
reasonable user will expect that additional patches might get applied
during the build.

Yes, we have to agree to disagree on that.
Post by Sean Whitton
Note that you were also arguing based on a different source
For example, someone might want to use a Debian system to investigate a
bug on an Ubuntu system. They might begin by downloading some source
packages from the Ubuntu mirrors. Since they obtained them from Ubuntu,
they will form the reasonable expectation that unpacking these source
packages will get them the code running on the Ubuntu system they are
debugging.
This would be useful for debugging problems.
But it is important to understand that in the general case there will
always be cases where the code running on your system will depend on
the architecture of your system - after applying patches the sources
might be architecture-specific.
Unless I'm missing something, that can only be true when the application
of patches to which you refer occurs during the package build.
dpkg-source -x --vendor=Ubuntu --arch=arm64 hello_2.10-1.dsc

--vendor should be implementable today based on vendor-specific patch series.
--arch would require similar support for arch-specific patch series.

I am not saying that a complete solution would be easy to implement,
or that this might happen anytime soon.

But the long-term goal should be to abolish patching during the package
build, by bringing more conditional patching functionality to dpkg.

This will allow packages to move away from their own custom patching
systems.

And it will give the user the sources that will actually get built.
Post by Sean Whitton
Sean Whitton
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Philip Hands
2018-08-19 20:13:56 UTC
Permalink
Adrian Bunk <***@debian.org> writes:

...
Post by Sean Whitton
"The source" is what you get after steps 1 and 2.
Why is "The source" what you get after dpkg applied patches,
but before debian/rules applied patches?
I agree with Sean's point about it being a matter of definition relating
to when we invoke debian/rules, but for an alternative justification one
might look at this:

For the Debian Maintainer, what is the preferred form of modification?

It could be the source before the patches are applied (especially if
they're working on a patch to be sent upstream), but really, chances are
that it's actually the state of the source after the Debian patches are
applied.

It is almost certainly _not_ the state that source might get transformed
into at some point during the build process.

It is also almost certainly not the alternative version of the source
that results from applying a patch series that only gets applied if they
unpack the source on a different vendor's OS.

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY
Adrian Bunk
2018-08-19 21:15:00 UTC
Permalink
Post by Philip Hands
...
Post by Sean Whitton
"The source" is what you get after steps 1 and 2.
Why is "The source" what you get after dpkg applied patches,
but before debian/rules applied patches?
I agree with Sean's point about it being a matter of definition relating
to when we invoke debian/rules, but for an alternative justification one
For the Debian Maintainer, what is the preferred form of modification?
The maintainer might actually know how his self-written patching
machinery works.

I am doing plenty of bugfixing in packages I do not maintain, and the
complex build machineries of some packages can be a real timeconsuming
pain in some body part.

It is also frustrating when you end up spending hours trying to find a
bug in the wrong sources.
Post by Philip Hands
It could be the source before the patches are applied (especially if
they're working on a patch to be sent upstream), but really, chances are
that it's actually the state of the source after the Debian patches are
applied.
So for src:gcc-8 the preferred form of modification are three tarballs
that are not even unpacked, and none of the patches applied?
Post by Philip Hands
It is almost certainly _not_ the state that source might get transformed
into at some point during the build process.
Why not?
It gets transformed into the sources that actually get built.
These are the relevant sources when debugging some FTBFS or segfault.
Post by Philip Hands
It is also almost certainly not the alternative version of the source
that results from applying a patch series that only gets applied if they
unpack the source on a different vendor's OS.
"different vendor's OS" is only a small part of the general problem.

How should we handle architecture-specific patches properly inside
Debian? This is part of the same general problem that the notion
of exactly one "the sources" is misguided when it comes to what
actually gets compiled.

It is a real problem that there is no standard way to look on an amd64
machine at the src:gcc-8 sources that will actually get built on armel.

And I cannot blame the maintainer, since there is no standard way to
express things like e.g.:
- "apply the Linaro patches only on arm* and only in Ubuntu"
- "apply the Hurd patches that might touch generic code only on Hurd"
Post by Philip Hands
Cheers, Phil.
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Wouter Verhelst
2018-08-20 07:28:30 UTC
Permalink
Post by Adrian Bunk
How should we handle architecture-specific patches properly inside
Debian?
Why should there ever be architecture-specific patches?

I get that there sometimes need to be vendor-specific patches, because
defaults may differ between distributions. But why on earth should
defaults differ between architectures? That just makes no sense. Things
like uintXX_t and htonl() should take away most architecture-specific
differences, and then all that remains are things like ensuring
alignment is done right. You don't need patches for that; you need
bugfree code for that.

I think, however, that "changing the defaults to make sure it matches
the one for this vendor" should be done as part of the build, if at all.
Doing it as part of unpacking the source is just wrong. When I run
"dpkg-source -x", I expect it to behave as would "unzip" or "tar x". To
have it act differently depending on the environment in which it is run?
That's just perverse.

[...]
--
Could you people please use IRC like normal people?!?

-- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
Hacklab
Adrian Bunk
2018-08-20 13:51:30 UTC
Permalink
Post by Wouter Verhelst
Post by Adrian Bunk
How should we handle architecture-specific patches properly inside
Debian?
Why should there ever be architecture-specific patches?
I get that there sometimes need to be vendor-specific patches, because
defaults may differ between distributions. But why on earth should
defaults differ between architectures? That just makes no sense. Things
like uintXX_t and htonl() should take away most architecture-specific
differences, and then all that remains are things like ensuring
alignment is done right. You don't need patches for that; you need
bugfree code for that.
...
Are we discussing bugfree code or are we discussing reality?

As an example, the non-release architectures of Debian are either
brand-new (riscv64) or fringe with usually brittle upstream status
in the toolchain (the other 13 non-release architectures).

Now look at
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85412

"Keywords: deferred" means "This bug was deemed too risky to attempt to
fix during stabilization stages. Deferred to a development stage of a
subsequent release." AKA "Target Milestone: 9.0"

The bug seems to be rare enough on x86 to warrant that.

On ia64 this bug breaks most code that builds with -O3,
Qt and Pyhon 3.7 were among the first FTBFS.

The fix will likely be in generic code in gcc.

Should the gcc maintainer apply a backport of a non-architecture
specific change to the compiler used in the buster release, if upstream
considers it too risky for the gcc 8 branch[1] and the fix is important
only for a non-release architecture?

Applied only on ia64 the situation is different.
On ia64 the alternative would be a lot of FTBFS bugfixing
work with workaround patches submitted to a three digit number
of packages.

And this can happen even on release architectures in a release.
In src:gcc-6 the fix for #727621 [2], which is required for
building LLVM on armel, is applied only on armel in stretch.

cu
Adrian

[1] perhaps the fix for this specific bug will turn out to be simple,
but let's assume there will be a fix and it will not be backported
to the upstream gcc 8 branch
[2] https://sources.debian.org/src/gcc-6/6.4.0-20/debian/patches/pr64735.diff/
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
Wouter Verhelst
2018-08-20 22:20:51 UTC
Permalink
Post by Adrian Bunk
Post by Wouter Verhelst
Post by Adrian Bunk
How should we handle architecture-specific patches properly inside
Debian?
Why should there ever be architecture-specific patches?
I get that there sometimes need to be vendor-specific patches, because
defaults may differ between distributions. But why on earth should
defaults differ between architectures? That just makes no sense. Things
like uintXX_t and htonl() should take away most architecture-specific
differences, and then all that remains are things like ensuring
alignment is done right. You don't need patches for that; you need
bugfree code for that.
...
Are we discussing bugfree code or are we discussing reality?
As an example, the non-release architectures of Debian are either
brand-new (riscv64) or fringe with usually brittle upstream status
in the toolchain (the other 13 non-release architectures).
Now look at
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85412
"Keywords: deferred" means "This bug was deemed too risky to attempt to
fix during stabilization stages. Deferred to a development stage of a
subsequent release." AKA "Target Milestone: 9.0"
The bug seems to be rare enough on x86 to warrant that.
On ia64 this bug breaks most code that builds with -O3,
Qt and Pyhon 3.7 were among the first FTBFS.
The fix will likely be in generic code in gcc.
Yes, but there's a huge difference here.

Compiling code for a given platform is not the same thing as running on
said platform. If the compiler runs correctly, then it doesn't matter
which platform it runs on; it's still the same compiler.

The architecture-specific issues that I was talking about, when they
occur in a compiler for a platform, can be patched perfectly safe, with
or without the agreement from upstream. If you see that a compiler
misbehaves when compiling *on* platform X but not when compiling *on*
platform Y (but in both cases compiling *for* platform Z), then you need
to fix the code on all platforms -- the ones where it is broken as well
as the ones where it is not broken -- and ship the fixed code. Hopefully
to upstream too, but definitely to Debian.

The architecture-specific code in a compiler that occurs when compiling
for a given platform needs to be built into *any* compiler which
produces code for that platform; so, not just in src:gcc-X, but also in
src:gcc-X-cross and src:gcc-X-cross-ports.

So even if we were to add code to dpkg-source to support
architecture-specific patches (which we don't currently and which I
think we definitely shouldn't), then this still wouldn't help the GCC
packages in Debian.

[...]
Post by Adrian Bunk
And this can happen even on release architectures in a release.
In src:gcc-6 the fix for #727621 [2], which is required for
building LLVM on armel, is applied only on armel in stretch.
No; it is applied only on compilers that produce armel *output* in
stretch.
--
Could you people please use IRC like normal people?!?

-- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
Hacklab
Yves-Alexis Perez
2018-08-20 18:54:22 UTC
Permalink
Hi,

I'm maintaining two packages which have Ubuntu-specific patches, which have
been imported by Ubuntu maintainers giving a hand here in order to reduce the
diff with Ubuntu and maintain the packages in the same place.

I don't have strong opinions on this, but if it's suddenly forbidden, someone
(not me) has to find a way that works for derivatives and doesn't end up
multiplying sources packages, because I think we'll lose in the end.
Post by Wouter Verhelst
I think, however, that "changing the defaults to make sure it matches
the one for this vendor" should be done as part of the build, if at all.
Doing it as part of unpacking the source is just wrong. When I run
"dpkg-source -x", I expect it to behave as would "unzip" or "tar x". To
have it act differently depending on the environment in which it is run?
That's just perverse.
I have read this multiple time in this bug. To be it looks like you're
advocating dpkg-source *not* to apply any patches to upstream sources, then.
To be honest, I think I liked it better when patches were actually applied
during the build and not during source extract.

Regards,
- --
Yves-Alexis
Tollef Fog Heen
2018-11-05 18:44:21 UTC
Permalink
I call for votes on the following resolution with regards to #904302.
The voting period lasts for one week or until the outcome is no longer
in doubt (§6.3.1).

==== RESOLUTION ====

Vendor-specific patch series are a feature of dpkg that can be used to
apply a different series of quilt patches when the source package is
unpacked on different systems. Since Debian source packages are usually
treated as a pure transport format (like tar), this property can cause
confusion and frustration for users. Examples could be if only the
series file for one vendor is updated, or a source package is unpacked
on one system and then transferred to a system with a different vendor
for debugging.

The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done by using differing source packages, or as part of the build
process using current and future practices such as patches with
conditional behaviour or patching of files during the build rather than
at source unpacking time.

Since this feature is used by several packages today, we need a
reasonable transition period. They will be considered buggy from when
this resolution is accepted, but it will not be considered severe enough
to warrant immediate removal from Debian. After Buster is released, the
presence of a vendor-specific patch series will be a violation of a MUST
directive in Debian policy.

The Committee therefore resolves that:

1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free).

This should be implemented in Debian Policy by declaring that a
package SHOULD NOT contain a non-default series file.

2. After Buster is released, use of the vendor-specific patch series
feature is forbidden in the Debian archive.

This should be implemented in Debian Policy by declaring that a
package MUST NOT contain a non-default series file.

==== END OF RESOLUTION ====

A: Approve resolution, disallowing the use of dpkg's vendor series
F: Further Discussion

- --
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Tollef Fog Heen
2018-11-05 19:18:11 UTC
Permalink
]] Tollef Fog Heen
Post by Tollef Fog Heen
A: Approve resolution, disallowing the use of dpkg's vendor series
F: Further Discussion
I vote A > F.

- --
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are
Didier 'OdyX' Raboud
2018-11-06 09:40:36 UTC
Permalink
Post by Tollef Fog Heen
==== RESOLUTION ====
Vendor-specific patch series are a feature of dpkg that can be used to
apply a different series of quilt patches when the source package is
unpacked on different systems. Since Debian source packages are usually
treated as a pure transport format (like tar), this property can cause
confusion and frustration for users. Examples could be if only the
series file for one vendor is updated, or a source package is unpacked
on one system and then transferred to a system with a different vendor
for debugging.
The Committee recognises that there is a need for packages to behave
differently when built on different distributions, but this should be
done by using differing source packages, or as part of the build
process using current and future practices such as patches with
conditional behaviour or patching of files during the build rather than
at source unpacking time.
Since this feature is used by several packages today, we need a
reasonable transition period. They will be considered buggy from when
this resolution is accepted, but it will not be considered severe enough
to warrant immediate removal from Debian. After Buster is released, the
presence of a vendor-specific patch series will be a violation of a MUST
directive in Debian policy.
1. Any use of dpkg's vendor-specific patch series feature is a bug for
packages in the Debian archive (including contrib and non-free).
This should be implemented in Debian Policy by declaring that a
package SHOULD NOT contain a non-default series file.
2. After Buster is released, use of the vendor-specific patch series
feature is forbidden in the Debian archive.
This should be implemented in Debian Policy by declaring that a
package MUST NOT contain a non-default series file.
==== END OF RESOLUTION ====
I vote A > F

Thanks for writing this resolution down!

Cheers,
OdyX
Gunnar Wolf
2018-11-06 15:35:43 UTC
Permalink
Post by Tollef Fog Heen
==== END OF RESOLUTION ====
A: Approve resolution, disallowing the use of dpkg's vendor series
F: Further Discussion
I vote:

A > F

Thanks,
Philip Hands
2018-11-07 12:14:09 UTC
Permalink
Tollef Fog Heen <***@debian.org> writes:
...
Post by Tollef Fog Heen
A: Approve resolution, disallowing the use of dpkg's vendor series
F: Further Discussion
I vote A > F

Cheers, Phil.
--
|)| Philip Hands [+44 (0)20 8530 9560] HANDS.COM Ltd.
|-| http://www.hands.com/ http://ftp.uk.debian.org/
|(| Hugo-Klemm-Strasse 34, 21075 Hamburg, GERMANY
Niko Tyni
2018-11-09 18:54:15 UTC
Permalink
Post by Tollef Fog Heen
A: Approve resolution, disallowing the use of dpkg's vendor series
F: Further Discussion
I vote: A > F
--
Niko Tyni ***@debian.org
Chris Lamb
2018-11-18 17:32:49 UTC
Permalink
With five votes in favour and none against, the resolution passes.
Thank you for your work on this. This has been implemented in
Lintian here:

https://salsa.debian.org/lintian/lintian/commit/256c8713d722ae56d887a30b222ecf4a684909bf

This could allow it become autorejected by dak once this code hits
stretch-backports.


Regards,
--
,''`.
: :' : Chris Lamb
`. `'` ***@debian.org / chris-lamb.co.uk
`-
Debian Bug Tracking System
2018-11-13 18:15:03 UTC
Permalink
Your message dated Tue, 13 Nov 2018 19:12:53 +0100
with message-id <***@err.no>
and subject line Re: Bug#904302: Call for vote on disallowing the use dpkg's vendor series in the archive
has caused the Debian Bug report #904302,
regarding Whether vendor-specific patch series should be permitted in the archive
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)
--
904302: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=904302
Debian Bug Tracking System
Contact ***@bugs.debian.org with problems
Sean Whitton
2018-11-17 16:40:03 UTC
Permalink
Hello,
The technical committee was asked in bug #904302 to decide whether to
allow the use of vendor-specific patch series in the Debian archive.
Thank you all for your work on this. Should be in the next Policy
release.
--
Sean Whitton
Loading...