Add repository caching and enqueue clone on startup#721
Add repository caching and enqueue clone on startup#721Kobzol merged 8 commits intorust-lang:mainfrom
Conversation
| let log_repo = repo_name.clone(); | ||
| match senders.gitops_queue().enqueue_clone_repository(repo_name) { | ||
| Ok(true) => {} | ||
| Ok(false) => { |
There was a problem hiding this comment.
Hmm, in practice this won't be an issue, as the current queue capacity is 3, and we only work with rust-lang/rust in production, but the capacity shouldn't be used for some commands. Let's not deal with that now.
|
I did some experiments on a repository that has the full history of It seems that there is some unfortunate git behavior when a bare clone is combined with a blobless clone. It is quite slow on the first push. The second push is fast again, but that's not enough. It seems to me that the sweet spot is either doing a bare clone with all blobs or trees, or a treeless bare clone, which saves space on disk, has fast init, relatively fast first push, and fast followup pushes. That being said, the treeless clone could cause some performance edge-cases, and I'm kinda worried about their behavior with submodules. We already saw some issues with partial clones on I took the liberty of adding some logging, so that we can observe in logs how long does the init and squashes take. Let's try to use the treeless clone and see what happens in production. |
src/bors/gitops.rs
Outdated
| std::fs::remove_dir_all(repo_path) | ||
| .context("Cannot reset repository cache directory")?; | ||
| } | ||
| if let Some(parent) = repo_path.parent() { |
There was a problem hiding this comment.
I don't think that this is needed, as git creates the target directory and its parents recursively when cloning.
2 participants
close: #693
Add gitops cache with
CloneRepository. and reuse cached bare repos for squash commit transfer.