Create non-email method of contributing kernel patches


 * Summary: Create non-email method of contributing kernel patches
 * Proposer: Tim Bird

Description
A consistent problem for "industry" developers is problems using their e-mail client for submitting patches to mainline. In particular, many industry developers work for companies whose IT policies dictate that they use Outlook as their email client and Exchange as their email server. These two programs are problematical to use for submitting patches to the kernel. They sometimes mess up the text formatting of the patches. Often, Exchange will be configured to automatically add confidentiality notices to outgoing email. These notices conflict with the intent of publishing the patch, and emails containing such notices will not be applied by upstream maintainers.

The purpose of this project is to create a method of submitting patches to the mainline Linux kernel community that is independent of any particular e-mail client or configuration, and that thus does not suffer from the problems mentioned.

One alternative to using one's corporate email account for submitting patches is to set up a separate email account (quite often a gmail account), and submit patches from that.

The primary motivations for not just using gmail accounts are:
 * it may be desirable to have the author of the patch attributed to the corporate email address
 * it may be difficult to reach gmail (or some other external email client) from inside the corporate firewall

Ideally, this feature would be integrated into the git command suite, which is the preferred tool for working with Linux kernel source. This may be as simple as making a "git- " version of this, and putting it in the right place where git can find it. (It appears the standard git exec-path is /usr/lib/git-core). It might be possible to just add this functionality to git send-email (maybe add a flag to indicate sending the "e-mail" via a web proxy or something).

what about follow-on emails?
Note that this project is limited in scope to just the submission of patches to the community mailing lists and addresses. It does not address the further problem of responding to feedback on the patches, from those corporate-mandated email programs. That is, Outlook doesn't handle inline commenting properly, and either Outlook or Exchange may mangle the text for emails related to patch reviews. For the purpose of this project, this is considered a secondary consideration, as it is possible to use gmail or some other email account external to the company to respond to feedback.

discussion from kernel summit list
Darren Hart and others discussed this topic on the kernel summit discussion list prior to the 2015 kernel summit. Some attributes of this tooling would ease the job of maintainers, and some would ease the task for commercial developers.

Here are some features that were mentioned that would be nice to have:
 * feature available as a git command

[under construction]

Areas of research
Is it possible to specify the author of the patch when e-mailing from a different account than the author's e-mail account? That is, what does git use as the author for the commit meta-data?

List of things that go wrong
Here is a list of common problems encountered when someone sends an e-mail from a non-community-compliant e-mail client: (below each one is how to fix it)
 * patch is sent as an attachment
 * patch is sent as Content-Type: text/html
 * needs to be converted to text/plain
 * patch is sent with Content-Transfer-Encoding: base64
 * convert to plain text with uudecode or base64?

Here are problems I have:
 * mis-typing a person's e-mail, and having the mail go out with a bad CC: This is extremely embarrassing, because everyone who responds to the e-mail gets a bounce
 * sending to the wrong people. Twice I left off the actual maintainer of a driver I was sending a fix for, because I misread the output of get_maintainers.pl.  This was also very embarrassing.

proposed features

 * check that destination addresses are valid (list names spelled correctly, individuals are in MAINTAINERS file or in a git commit, or in the senders address book)
 * check the list against the addresses listed by get_maintainers.pl
 * run scripts/checkpatch.pl against the patch
 * use the mail threading options appropriate for the primary target of the patch
 * each list or maintainer has their own preferences for some submission methods - read those from MAINTAINERS and implement as desired
 * allow user to specify the author e-mail address of the patch
 * Needs some validation that they own the address, lots of abuse potential here...
 * Check that the patch is against a sensible kernel version.

Related work

 * See Overcoming IT Obstacles to open source participation
 * git send-email
 * RFC5321 - Simple Mail Transfer Protocol
 * There was discussion of this on the ksummit-discuss mailing list (in preparation for the 2015 kernel summit)
 * Darren Hart mentioned email tooling here: http://lists.linuxfoundation.org/pipermail/ksummit-discuss/2015-July/001417.html
 * Josh Triplett mentioned some more tooling features here: http://lists.linuxfoundation.org/pipermail/ksummit-discuss/2015-July/001417.html

Outlook inline comments
Here is some information on how to configure outlook to use "> " to preface replied-to text in inline responses.
 * https://support.office.com/en-us/article/Reply-with-inline-comments-within-the-original-message-text-5bc44105-aeb2-49e2-8239-5aeeff52e5a7
 * Note that you set "When Replying to a message" to "prefix each line of the original message", and you set "When forwarding a message" to "prefix each line of the original message".

These instructions worked for me on Outlook in Office Professinoal Plus 2013 * click on File on the menu bar * select Options on pane that appears * select Mail on left pane * under Compose Messages, select change the setting for "Compose messages in this format" to "Plain text" * Under Replies and Forwards * Set "When replying to a message" to "Prefix each line of the original message" * Set "When forwarding a message" to "Prefix each line of the original message:  * Set "Preface each line in a plain-text message with:" to "> "  (greater-than sign followed by a space)

The above instructions work when the message is received in plain text. When a message is received in HTML format, it appears to be impossible to do inline or post-message comments.

From Andrew Ducker at http://superuser.com/a/587558/597172

You can cause Outlook to format all inbound mail at plan text as follows: * File * Options * Trust Center * Trust Center Settings * Email Security * Read As Plain Text.

This means they all appear as plain text, so replies go out as plain text, which means that you get proper indenting (using ">") if you've set that up.

You can still display the HTML original for messages where you want to, by clicking on the "This message was converted to plain text" message on the header and switching to a different format.

Tim - Note that once you select the option to "View as HTML", you cannot switch back to plain text again. (At least, I couldn't find an option to do it. The info box that appears no longer has any options for changing the view formatting.)

In Outlook, if you reply, you may be put in a composer pain in the main outlook window, or it may create a new window to compose your response. In the standalone window, you can control the output format by selecting: * Format Text * in the "format" section, select "Aa Plain Text"

If it starts your resonse composition in the main outlook window, there is an option to "Pop Out" the message into a standalone window. If you want to always start responses in a standalone window, set the option: * File * Options * Mail * Replies and Forwards sectino * check Open replies and forwards in a new window There's an option to prevent Outlook from removing extraneous line feeds. See * File * Options * Mail * Message Format * Remove extra line breaks in plain text messages

Note that in that same section of options, there is control over the column to wrap plain text at: * Automatically wrap text at character [  ]

There's an way to prevent the first letter of a new line from being automatically capitalized. Change the autocorrect setting: * File * Options * Mail in left-side menu * Compose Messages (section) * Spelling and Autocorrect button * Proofing in left-side menu * AutoCorrect options section * AutoCorrect Options button * AutoCorrect tab * uncheck Capitalize first letter of sentences

Scope
Describe the expected amount of work (in person-weeks, please)

Contractor Candidates
List possible contractors for this work (can be yourself)

Comments
Some random ideas: for a response to send the email forward to the lists?
 * is it possible (or advisable) to use the list of e-mail addresses already in the kernel git tree to validate new incoming e-mails
 * idea is to encourage more newcomers. Maybe if the from address is already valid, then eliminate some validation tests?
 * need to prevent spamming the lists. Do some kind of challenge response, where an e-mail is sent to the from address, and wait
 * this is the kind of thing that should be automate-able if the person already has a patch in the kernel tree. Otherwise, use some other method of spam avoidance.
 * If we just accept things for already existing e-mails then people will just abuse that...

stuff about outlook and exchange
From this LWN.net comment

Thunderbird+Lightning+EWS Provider appears to be a decent Outlook alternative if you're stuck with Exchange on the backend. It's far more stable and performant than Evolution
 * For information about EWS Provider, see https://github.com/Ericsson/exchangecalendar/wiki/Exchange-EWS-Provider
 * For info about the lightning plugin, see https://addons.mozilla.org/en-US/thunderbird/addon/lightning/

ideas from Darren Hart and Mark Brown
Darren wrote (in a private e-mail in May, 2016):

1) I would decouple this from git initially (maybe permanently). First, demonstrate it works without having to talk to the incumbents! Make it consume existing output and use existing transports, but present an accessible interface to the newcomers.

2) Other challenges include proxies which could complicate git usage.

3) New users have a lack of advanced tooling (static analysis, etc.), all of which can be used to increase patch quality before it ever gets to the highly contended resources like reviewers and maintainers.

4) The service should require email. I think partnering with something like gmail through the google auth mechanism would be a good way to go. It's not overly difficult to convince gmail to only send plain text email, so anyone that can access the web tool to submit the patch, should also be able to discuss the patch via email. (this is another issues to address - rejecting multipart email when it includes HTML but also includes plain text is a problem that hurts our productivity IMHO which is shared by some senior kernel devs as well - but, that's a separate rant)

5) Additional errors: * checkpatch (reject on error, warn on warnings) * spelling (good grief, but yes) * untested build failure (config randomization, architecture, etc.)

6) Integration with 0-day will help with 3 and 5

and, later:

I think it's worth bringing up on kernel summit discuss. I received less push back on this with folks like Greg KH than previously when we revisited it at an Intel event in March. Still, I suspect there is too much of a fixed mindset among senior maintainers, and it won't directly change anything that group does daily (I would say it cannot - they need to continue to scale). I expect we'll have to get this up and running and then share results at ksummit 2017 - and hopefully the results are good!

and

What I have been discussing since KS'15 was a web tool that new developers could submit patches through. It would accept a patch (git format-patch -1) or a zip/tar of a series (git format-patch -N && tar cf patches.tar.bz2 000*.patch), verify formatting and checkpatch (fail on error, warn on warnings), apply them to master (or possibly the selected subsystem maintainer's tree), check for recipients (all people and lists and from MAINTAINERS), and possibly add it to a known branch which 0-day can pick up and do a build on, only then will the tool present the user with the "submit for review" button.

Mark wrote:

I added a couple of requirements to the wiki, verification for author addresses (otherwise there's abuse potential) and checks to make sure people are using a sensible kernel tree as a base.

(regarding (1) above)

I think if people are not using git already that's a whole extra world of pain - things like getting the right tree to generate the patch against will be fun. From that perspective I think Tim's suggestion is good here, get something where you can have your patch sitting in git and work directly from there rather than having to export it into some other tool.

This is a thing where we've already improved the tooling a lot, before git formatting things properly had a lot more gotchas but now there's a bunch of tools in git which can get patches out in sensible and robust formats.

(regarding (2) above)

I think the suggestion here is a tool that presents on the command line in an integrated way with the rest of the git UI rather than specifically using the git protocol?

(regarding 4 above) Blocking public mail providers like gmail is relatively common for stricter corporate IT setups so making it a requirement becomes tricky for some of the intended users - if gmail and whatnot are OK then most things should be fine already, it's pretty well documented how to use them already. Probably something more traditional like setting up an account and then using a verification e-mail to confirm the address would be needed (but of course more direct integration with providers like gmail would be useful to some).

From that perspective one thing we might want to consider here is providing any submission tools as a locally installable thing (perhaps including packaged VMs) so corporations can install it on their systems with whatever auditing and lockdown they desire. That would both allow them to make setup easier for their users (eg, integration with SSO systems rather than needing an external account) and have a greater feeling of control over what's going on.

and, later:

Perhaps another thing someone could work on in parallel is a tooling for helping people verify that their e-mail is coming out sensibly? Your idea of a test list from the ksummit thread was a good one.

Darren wrote about minimizing the need to install something on the developer's system:

Many developers work on Linux systems in a lab with very restricted access, so their main client is Windows or a Mac. We could run Docker or similar on these systems, but some developers don't even have administrative rights on their systems to install software. It would still require proxy configuration - which is one of the biggest issues for new OSS devs at Intel (for example).

And about the usability of "git send-email":

Erm... "git send-email" is more like exactly the kind of tool I think is the problem. It requires quite a bit of client setup to get working correctly and has a number of non-obvious configuration options which will get you yelled at if you get them wrong.