Repository navigation
Time to address the elephant in the room: There are too few users with PR merging rights who merge PRs. #333858
Replies: 17 comments 51 replies
|
Imre Eilertsen (@DandelionSprout) We're at an inflection point in terms of the number of PRs being submitted and the ability for the infrastructure to gracefully handle the load of automated validation. We have some large changes in progress: The project gives a view of items currently deemed "important" to work on next. I've had to make some tough calls relative to the number of Microsoft folks who have PR merge rights and should focus on manual reviews vs. improving the infrastructure. Right now, I'm optimizing for getting these big changes implemented and released. I see this issue as more of a "call to action" or a "discussion" as opposed to a feature or bug to be implemented. I'm going to go ahead and convert it to a discussion. |
I guess you're technically right about that, now that you mention it. |
|
Most of our "moderators" are community members. They have the ability to "moderator approve" PRs that have passed the automated validation steps. They don't necessarily have all the insights into what went wrong when something doesn't just flow through the system. There is only so much they can do, and they are volunteering their valuable time and knowledge. Handling the exceptional cases (I think of these often as "known classes of problems") often requires someone from Microsoft to review. These are where the most time is consumed for people working on the project. The following Issue has a list of known classes of problems: We see those as areas where it takes more "human" time that generally extend the average time from PR submitted until the package version has been published. We will be prioritizing the ones that have the biggest impact in terms of the amount of time that class of problem impacts our overall throughput. We've been doing a lot of work behind the scenes to improve visibility for what went "wrong" or why a PR "failed" automated validation. When we have our next big release with the GitHub App and a few other improvements in progress (like queueing) we're expecting to reduce the time it takes to investigate. It should also surface more information directly on the PR in the form of comments as opposed to requiring someone to go spelunking in log files. |
So do I for sure.
I approve of that idea. After all, the automated pipeline comments are usually things like (paraphrased) "We unfortunately couldn't install this silently. Please do some changes and submit again.", while
|
|
Exfiltrating logs from a hardened environment requires extra scrutiny. We're working on making it easier to get to the logs, and we're working on more clear error messages to help understand the actual reason things failed. The "Internal-Error-Dynamic-Scan" is tantamount to a generic exception and is rarely helpful. The goal is to be able to explain in understandable terms what went wrong so it's easier for someone to understand what needs to be done. |
|
Just thinking out loud. I like winget. I think it's one of those things MSFT is still mostly doing right (compared to a lot of other divisions I could roast). However, the key thing I see here is a staffing ($$$) issue. Microsoft isn't committing the resources to winget which is an incredibly important project to holding up the broader security of the entire Windows platform. Microsoft, via winget, helps developers keep their software deployed to end user machines faster with the least amount of trouble. What's the security cost of NOT getting third-party vulnerabilities on Windows patched? Why is Microsoft not investing in the staff? Wouldn't staffing this project appropriately directly support a "secure by default" mission? Sounds to me like there is a direct parallel between this discussion and the very origins of Winget. Microsoft is cutting corners. |
|
Is there a reference for:
The REST API is something I decided to build so enterprise customers could stand up their own private sources to distribute private line of business packages and use as a curated set of approved packages. I never expected or intended to have a large set of sources. WinGet could support that, but other than developers, power users, and IT administrators are a relatively small number of users in the Windows ecosystem. CLI tools generally aren't very popular with consumers. The community repository wasn't a "thought experiment" or a "proof of concept". It was genuinely intended to provide a way for reducing the friction involved in distributing "developer tools". I've spoken about it several times at conferences (many of those are online). There is no shortage of future improvements or enhancements for both the "community" use cases or "enterprise" use cases I'd like to enable. One of the ones I'm digging deeper into currently is: |
Good to know.
Give me 20min and I'll see if I find it. |
|
I could've sworn that marketing that lead up to the release of Windows Server 2025 focused on that it was a way for companies to create their own repos to internally distribute/deploy apps approved/added by the companies' IT teams. But I haven't been able to find it again, and instead https://learn.microsoft.com/en-us/windows/package-manager/package/repository came up much more prominently when I searched around on Qwant and Google. As such, I apologise for my insufficiently founded claims. |
|
No worries. I would not have been surprised to see unique claims made by others. I would have contacted them to have a discussion. It took a lot of effort to have WinGet included in Windows Server. I tried for a few years until it actually became one of the top requested features for the Windows Server team. In the end, they actually approached me about what it would take in terms of product development, and fortunately we had already prioritized the work that could have prevented success. It can be very interesting getting things done in such a large organization. Many of the features being launched/released were years in the making (even features that are trivial in terms of software engineering). As an aside: I really appreciate the interest in helping make WinGet become the best product it can be. Discussions like this can be challenging, but I'd rather be open and transparent. I can also be convinced to change my mind, and even when I don't necessarily like it, I'll admit when I'm proven wrong (this happens regularly). I can only imagine how challenging it is to reason about what's going on with validation since it's not open source. The reality is that we're using a hardened infrastructure designed not to let PUA (Potentially Unwanted Applications) get out into the wild. We have to assume that every package we're validating contains nefarious code. That adds a bunch of complexity. We've come a long way from the days when I used to manually validate every single PR myself, or I had to find someone else to validate packages I submitted. This is just another one of those points in time where we're hitting growing pains. |
|
For anyone curious about developments in one of this subthread's fields, at least 1 third-party repo now exists: https://github.com/pl4nty/winget-extras |
|
This week more than ever have I begun to realise a particular thing: Surely it can't be good for a repo to have 0 admins available on weekends? |
|
Yet another factor in play: The lack of admins, also makes this repo unable to handle (and probably ban) mass spammers/trolls/bullies/harassers within the timespans where those would need to be handled; see #351271. Edit: I've @-ed 5 admins (and 1 mod) there thus far, and the only thing on my mind right now is: Where are everyone? There's a 23-strong moderator list at https://github.com/microsoft/winget-pkgs/blob/master/doc/Moderation.md, but none have at the time of writing handled the most tremendous spammer in this repo's history. |
|
Just making a comment here. There have been additional cases of mass spammers. GitHub has some processes to handle them, and we have our own if those don't trigger. I've recently evaluated the contribution graph for the administrators (folks who work for Microsoft and have "write" access), and "active" moderators. I've also amended some of our policy and process documentation regarding them. Just because someone has "write" access doesn't mean they are actively watching everything. We have a small, but nimble crew of folks working on WinGet who are very passionate. Our community moderators do not have write access, but a few of them have means to contact some of the administrators directly. |
|
Stephen Gillie, though he certainly goes beyond the call of duty to handle the massive load of ≥600 PRs a day, seems very overworked from what I personally can tell at the time of writing. So surely now it's a suited time to add a 3rd active admin? Presuming I can't nominate myself, then Trenly and pl4nty come across to me as the most suited candidates in the immediate short-term. |
|
I appreciate the energy and enthusiasm you bring to the project! If it was as easy as just add another admin it would have already been done. I've had several discussions with folks about getting more support here. It takes time to build the business case and get the approvals necessary. Given the potential blast radius, it isn't something we take lightly. Only Microsoft employees or vendors can be authorized to merge PRs in this repository. Some of your comments have been included in the justification. I have actively been working on this and will continue to do so. |
If need be, I'd likely grab the opportunity if Microsoft wanted to hire me to handle things like these. /genuine. Though in the immediate short term, it'd probably be easier to get mdanish and some others to step up to plate for winget-pkgs stuff and not just winget-cli. |
|
It looks like someone else has noticed how efficient Stephen Gillie (@stephengillie) is: |
|
It's still unfair to have to rely in our entirety on him to be a superhuman. Sooner or later, by the nature of man, he'll get burnt out from overworking at this pace. So what's the plan B if that happens? The perplexing blunder that was #333364 (comment) makes me fear the burnout is coming sooner than later. |
|
A tally I'll keep (with high potential for updates) of which known existing admins/mods with merge rights (including ones that microsoft-github-policy-service is the de jure merger for), besides Stephen and denelon, that are responsive to @-s or not:
⁽¹⁾ = Lacks the needed rights to remove e.g. |
|
So, I did not consider the possibility that Stephen Gillie would be the one to remain active, while Denelon would be the one to be AFK. But it's what has happened in the past week as of at the time of writing. Do we have a Plan C for this? |
|
@stephengillie has already announced he won't be as active quite a while ago. 
|
|
While that is good to know, the current situation is that Denelon has been absent on GitHub for a week now. |
|
I took some time off. I'm back now. 😊 |
|
Also good to know. |
|
Honestly, I'm not sure how long a PR can be inactive before it's appropriate to @ the maintainer when there's a red label (known issue or already explained) and the PR is not especially urgent. Also, I noticed the
Manual-Validation-Completed
|
|
I've had packages without red labels sit around for weeks. And I've had red labels be re-added to PRs after they had initially been waived. And recently, I'm also convinced there's a secret moratorium on new packages, but which no one have confirmed. At this point I think people way higher up in the Microsoft Corporation leader chain than Denelon (We're talking CEO boards and stuff) are either: And that this leads Microsoft Corporation to not hire any additional admins. |
|
There is no moratorium. 😊 Regardless of the size of Microsoft, or the market cap, we live in a world of infinite wants and limited resources (so far). There are many activities that are required to deliver products and make changes to products at Microsoft (or any large company for that matter). Many of these things are going on behind the scenes and as such aren't directly visible here at GitHub. As much as I'd love to have the authority to just spend as much capital or increase resources for WinGet, I'm logically competing with every other project/product/initiative at Microsoft, and some of those other things I agree with. Security, legal compliance, localization, globalization, internationalization, and release management are just a few of the things on the list. |
|
Sidenote, because I know people are wondering by now, as to the scheduling I use for @-s and similar at the time of writing until further admins are hired by Microsoft Corporation: For my own PRs:
For others' PRs:
|
I did some searches, and it seems that when you force-push a PR so that it matches the commits on the target branch, GitHub will automatically close the PR. Reset the branch commit to the original PR's commit, then force-push again, this should be able to reopen the PR. Sadly, I couldn't find the relevant documentation.
|
|
Demitrius Nelon (@denelon) - I suspect the motivations that Imre Eilertsen (@DandelionSprout) wrote apply to myself and many others - But most simple we want/need to install something, and wish it were available. There's of course a bit of the "nice to contribute something that others may want and not duplicate effort", or to support a specif project. I've contributed the occasional package manifest here, some to (Mac)brew, mostly for the same reasons. In my $Dayjob, where proper config management isn't an option (interfacing w/ the departments that manage Group Policy and the like is a non-starter, DSC isn't a great option for various reasons) WinGet is becoming the core of our VM install/config/update scripts. Double Thank You to whomever implemented the winget Powershell Module! (And even with it shipping in Server 2025 there's still about 30 lines of Powershell script I have to run on each one to get it properly inited and working.) So even though I'm essentially doing some unpaid work to provide "value" for a behemoth like Microsoft, it's so far worth it. Please communicate to whomever that it's a ridiculously overdue need. |
The sole mildening factor when it comes to Microsoft Corporation's increasingly running-ashore handling of Winget, is the fact that in sharp contrast to other ultrabig corps like Google and Apple, which really are centralised megacorps, Microsoft consists of perhaps 1,000 smaller teams, almost none of which ever communicate with each other. That's how Microsoft has ended up with no less than 5(!) package managers: Winget, Nuget, Powershell Gallery, vcpkg, and Microsoft Linux Package Repositories. And also that if someone managed to get a hold of the highest-up CEOs who worship and deify Copilot on a spiritual level, and asked them about any of those 5, they'd almost certainly go "…We have package managers?" |
And Store CLI? |
|
Good point. That makes it 6. |
|
At the time of writing this, there are 2.5 PR's currently open and seems to just be going up. Any plan on how the surge is going to be managed? |
As someone who submit lots of new packages, the situation is already slowly getting a scent of potential rioting among frequent submitters of new packages. Average processing times have now gone from "2-4 weeks" to "No longer able to be calculated due to the sheer lengths of time involved". |
|
Sidenote that I couldn't figure out how to fit well to the comment chain above: There's been a rush of amazingly bad PRs in pkgs in the past couple months, primarily from vibecoders. Creating a rule like "You as an app dev cannot submit a PR for your own GitHub app, unless it has ≥3★, its repo has been created ≥365 days ago, or both", would've instantly be able to close possibly as many as 900 of the currently open PRs. |
I agree. I don't think this is a solution. Even as a very infrequent submitter just of my own apps so far, I definitely feel things aren't working quite right.
I think this could backfire, or risk unfairness. I have a PR stuck in the waiting state right now for an app of my own which is (at least for now) closed source, so not in GitHub or having the opportunity for measurement based on stars. Under a rule like this, would I be prioritised over my equivalent self had I made it open on GH? Or would we be in effect closing things down and saying - despite past messaging - that certain apps are no longer welcome in this repo/package source? Maybe I'm being unfair in saying this, and I absolutely do not want to trivialise the issue or undermine the previous and ongoing hard work of everyone. But I feel like an organisation the size of Microsoft should be able to figure out a solution to this which works. Or provide a solution which removes the reliance on this one centralised repository. As someone previously mentioned, other solutions like Homebrew and Scoop allow for standing up their own repos/sources/whatever rather than everything being reliant on just one central source of truth. |
The way I thought about the idea: Yes. 😅
So do I.
Winget does technically allow custom sources, but the requirements to make a custom source are so steep that fewer than 10 people have got custom repos publicly accessible. The biggest breakers are that custom sources cannot be hosted on git sites, and that they must be hosted on a server with a web domain that the source owner has a valid SSL certificate for. |
Yes that's true. I was imprecise with what I said before - I should have said "allow for easily standing up their own repos/sources/whatever". Something with a similar level of approachability as those tools. Scoop just relies on a Git repo as the source, for example, which is super easy, and adding a new source (bucket) as an end-user is equally easy to do. Obviously I'm talking about potentially quite a big change to WinGet itself to support this kind of thing, so it may be infeasible. But looking at it from a high level, I think there's either the problem of making the current approach viable, or replacing the current approach with something more distributed-by-design. The lack of documentation on doing it as things current stand also seems to speak to the expectation that this isn't the "blessed" way. That might be a me problem if I've just missed it, but in addition to that I'm sure I remember at least one comment (possibly from Demitrius Nelon (@denelon) although I might be wrong, apologies if I am) saying that this is the case and that custom sources were only really envisioned for large enterprises, not for the more general case. |
|
another side note: some package requests or repositories are simply... insane, or their release cycle is... taking a look at the repository behind that new package request - seems to me that their AI release-management bot is publishing multiple releases DAILY.. sometimes another new release every few hours. That's no longer development for releasing stable versions... it's looking more like a command & control architecture that uses GitHub for deploying rapid mutations ... and their AI bot (or someone working with them) even requested inclusion in WinGet packages. * facepalm * |
|
It wouldn't exactly be the first time that someone who used Aİ to create the entirety of their apps, would also use Aİ to write their Winget-pkgs submissions. In fact I think well over 250 of the currently open PRs for new packages have done exactly that. |
|
but how many of those apps are pushing a completely new release version every few hours? |
|
Has this elephant been addressed yet? I see more and more people demanding review of their package (mostly on
New-Package
Is there really no solution to this? Something we can do at least? |
Unfortunately no. |
|
Well at least bring that back to the zoo |
|
well... the elephant is now in the zoo.... #436255 is there any chance to have it assigned a front stage and have it pinned at the top of the issues list? considering the volume of recent new package issues and PRs and that we're in the last part of September, it looks to me that WinGet inclusion has probably become part of class curriculum somewhere... and they didn't bother with creating a private WinGet source repository but they just want to stuff it in the main public one. |
|
With 2 analogical elephants for all to see in the analogical zoo exhibition, and the following on top of that: I know I've been doomsaying a lot over the past 10 months, about that pkgs was on the brink of absolute disaster. Some of the doomsaying was genuine and gloomy, whereas some of it was me simply hoping to lift/encourage overall performance. But now it really has gone past the brink. I'm pretty sure it's not viable to work through a PR backlog when Team GitHub literally doesn't want its users to get a proper overview of that backlog. So I'm cutting my losses and closing non-"Big deal" PRs of mine, and am hoping to diversify my hobbies beyond just Winget, Sims 3, and adblocking. It's not a goodbye, however, because if it had been one, I wouldn't have submitted e.g. #439905, as that one was not in my own semi-public backlogs previously. (Extra note about me and the 40-page limit:)I was in such astounded disbelief by Team GitHub's decision that I thought I was having a stroke, and was putting on autumn outdoor clothes in case of a potential hospital visit until the GitHub thread replies (and 1 Discord acquaintance) confirmed it wasn't just me. |
|
So now there are 3 elephants? |
|
i wonder how many elephants can the WinGet spider web handle... |
|
I agree that there is a real problem here, and I can see why some changes are needed. |
|
I know this repo has become "Lord of the Flies" and almost "Every man for himself", but surely it should be possible to put in place some sort of additional standards. For instance, it really should not be considered acceptable for a user to have submitted and keeping open 51 PRs at the same time (and counting) for the same new app: https://github.com/microsoft/winget-pkgs/pulls?q=is%3Apr+state%3Aopen+author%3Aswarupdonepudi |
|
as i write this, that link leads to 56 open PRs ... they squeezed a few more PRs in less than 24 hours, since you posted your message yesterday. That is quite an insane AI agent at work, one that has no concept of using a private winget source repository or what a stable version is supposed to be. |
|
And we're now at 58, might be worth pinging Swarup Donepudi (@swarupdonepudi) for this specific case. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
#333536 makes me think the time is right.
Essentially:
Validation-Completedtags.Validation-No-Executablesis the most severe example (with# 329209elaborating partially on it).# 332112.# 329263and# 333533).The 2 possible solutions are the same as in similar cases throughout GitHub where PR backlogs have got out of control:
All reactions