Do Git Compression and HTTP Buffer Settings Really Speed Up Clones?
Two settings intended to speed up CI were compared using public repositories on GitHub and GitLab.
core.compression = 0
http.postBuffer = 1073741824Each setting was tested separately and in combination using shallow clones over HTTPS with a full checkout.
Neither setting showed a consistent speedup. On both services, the received packfiles had identical sizes and SHA-256 hashes across configurations, as did the HTTP request counts and POST body sizes.
This article examines the Git documentation and the benchmark results.
Background
I came across articles recommending two ways to speed up CI checkouts: disabling compression and increasing the HTTP POST buffer. Their arguments were:
- Disabling compression reduces CPU usage and allows more data to be transferred over the network.
- Increasing the HTTP POST buffer reduces the number of round trips when transferring large repositories.
With settings like these, it is easy to confuse where they apply—the client or the server—and the direction of data transfer.
How Git Transfers Data
Objects, Loose Objects, and Packfiles
Git stores file contents, directory structures, and commits as objects. An object containing file contents is a blob.
Git primarily uses two storage formats:
| Storage format | Description |
|---|---|
| Loose object | Stores each object in a separate file |
| Packfile | Combines multiple objects into a single packfile |
Packfiles can use delta encoding to exploit similarities between objects, as well as zlib compression. core.compression=0 controls the zlib compression level; it does not disable delta encoding.
When preparing a packfile, Git may also reuse existing compressed data. It does not necessarily decompress and recompress every object.
How Clone and Fetch Work
In a typical Git transfer over HTTPS, the client requests the objects it needs, and the server prepares a packfile for transmission.
Client Server
│ │
├── Fetch request ────────────────>│
│ Prepare pack
│<──────────────────── Send pack ──┤
Receive, verify, store │
│ │git clone creates a new local repository. By default, it also checks out the selected commit after downloading the objects, writing files into the working directory.
git fetch downloads objects and references into an existing local repository. Fetching alone does not update files in the working directory.
Checkout also involves reading compressed objects and writing files. Therefore, the total execution time of git clone includes more than network transfer.
How Push Works
With git push, the client sends the necessary objects to the server.
Client Server
│ │
Prepare pack │
├── Send pack ────────────────────>│
│ Receive, verify,
│ and store
│<─────────────────────── Response ─┤Clone and push differ in which side prepares the packfile.
git config Changes Settings on the Machine Where It Runs
Running the following command on a CI runner changes the Git configuration for that runner’s user:
git config --global core.compression 0The --global option applies to the current user’s Git configuration. It does not send configuration changes to GitHub or GitLab servers.
For a self-hosted Git server, changing the configuration read by the Git processes running on that server can affect server-side behavior. The relevant configuration files and user accounts depend on how the server is deployed.
Users of GitHub.com and GitLab.com cannot change the hosting service’s compression settings through their local git config.
For temporary experiments, git -c allows you to specify a setting without modifying any configuration files.
git -c core.compression=0 clone https://example.com/group/repo.gitIn this example, the configuration is passed to the Git process and the Git subprocesses it launches.
What core.compression=0 Actually Changes
Default Compression Level
According to the official Git documentation, core.compression specifies the default compression level: 0 disables compression, -1 uses zlib’s default, and 1–9 offer different trade-offs between speed and size.
This value serves as the default for two other settings:
core.looseCompression: Compression level used when writing loose objects.pack.compression: Compression level used when writing objects into packfiles.
Explicit values for these settings take precedence. With all compression settings unset, loose objects default to level 1 and packfiles to zlib’s default.
How Compression Settings Affect the Client
On the client, compression settings can affect:
- Writing new loose objects with
git add. - Creating packfiles with
git gcorgit repack. - Preparing packfiles for push.
- Unpacking received packfiles and rewriting their contents as loose objects.
If compression accounts for a significant portion of execution time, lowering the compression level may reduce CPU usage. However, it may also increase storage requirements or the amount of data transmitted.
Changing the setting does not automatically recompress stored objects, as the Git documentation notes.
Client-Side Compression Settings Do Not Control Received Packfiles
The server prepares the packfile for a clone. The client’s core.compression=0 does not set the server’s compression level.
If the client stores the received packfile directly, the compression of the received objects remains unchanged. Decompression is still necessary when reading those objects.
If the client unpacks a received packfile into loose objects, it may recompress them locally. git unpack-objects extracts objects from a packfile and stores them as loose objects; local compression settings may affect this process.
The distinction is:
| Operation | Effect of client-side compression settings |
|---|---|
| Server prepares a packfile for clone | Client settings do not determine the compression level |
| Client writes new objects or packfiles | Local compression settings may affect the process |
Disabling compression on the client does not make clone transfers uncompressed.
What Happens If Compression Is Disabled on the Server?
Changing server-side compression settings can affect newly compressed data. Its impact should be evaluated in terms of CPU time, transfer size, and client-side decompression time.
Reusing existing packfiles or compressed objects also affects the results. Even if compression is disabled on the server, not every transferred object will necessarily be uncompressed.
The GitHub and GitLab server configurations were not changed and were not accessible for modification.
What http.postBuffer Actually Changes
HTTP POST Buffering
http.postBuffer sets the buffer threshold for POST data sent over Git smart HTTP. The default is 1 MiB (1,048,576 bytes).
The tested value, 1,073,741,824 bytes, is 1 GiB.
git -c http.postBuffer=1073741824 clone https://example.com/group/repo.gitClones also use POST requests to fetch objects, and the responses may contain large packfiles. But the two data sizes differ:
Client → Server: POST request body containing the fetch request
Server → Client: Packfile data returned in the responsehttp.postBuffer applies to client-to-server data, not to a download buffer holding the received packfile.
Data Larger Than the Buffer Can Still Be Sent
HTTP/1.1 supports chunked transfer encoding, which sends a body in chunks without knowing its total size in advance. The chunks belong to a single HTTP request.
Git documents using HTTP/1.1 chunked transfer encoding when POST data exceeds the buffer. HTTP/2 uses DATA frames instead. Neither requires a separate HTTP request for each chunk or frame.
When Should the Buffer Be Increased?
The Git FAQ suggests increasing the buffer for servers or proxies that mishandle chunked transfers. Normally, the default should be kept.
A large push may exceed 1 MiB, but transfers can proceed beyond the buffer threshold.
When Clone Requests Become Large
Clone and fetch POST bodies can grow large when they include many object IDs. Receiving a packfile of 100 MiB or several GiB does not mean the POST body is similarly large.
In the GitLab measurements, POST bodies were 375 and 201 bytes, while the received packfile was about 137 MiB.
A Large Repository Is Not the Same as a Large Push
Push size depends on the packfile needed to transfer objects missing from the destination, not just the repository’s total size.
| Operation | Factors affecting transfer size |
|---|---|
| Changing a few lines in a large repository | The push may be small if relatively few new objects are needed |
| Modifying every file | Depends on the number and size of objects and the effectiveness of delta compression |
| Replacing a large binary file | Transfer size may increase significantly if delta compression is ineffective |
| Initial push to an empty remote | May require sending many objects, including repository history |
| Pushing many commits at once | Depends on the amount of history and object data missing from the server |
Only clones were measured, not pushes, initial pushes, or pushes involving large changes.
Which Settings Affect Which Operations?
| Operation | core.compression=0 | Increasing http.postBuffer |
|---|---|---|
| Client-side clone | May affect local recompression, but not server-side packfile compression | Generally no effect |
| Client-side fetch | Same as clone | May matter if requests contain many object IDs or other data |
| Client-side push | Can affect compression when creating the outgoing packfile, although existing compressed data may be reused | May matter for large POST bodies, but transfers normally work with the default |
| Local add, gc, repack | Affects compression of newly written objects or packfiles | No effect without HTTP transfers |
| Server handling clone or fetch | Server-side compression settings may matter; client settings are not applied | Does not configure the server’s buffer for receiving client POST requests |
| Server receiving a push | May affect compression when the server rewrites objects | Does not specify the maximum accepted POST body size |
Benchmark Setup: GitHub and GitLab
Repositories
| Item | GitHub | GitLab |
|---|---|---|
| Repository | pytorch/pytorch | gitlab-org/gitlab |
| Fixed tag | v2.9.0 | v18.0.0-ee |
| Checked-out commit | 0fabc3ba448 | 504fd9e5236e |
Entries counted using git ls-files -z | 19,417 | 87,073 |
| Stored packfile size | 58,668,712 bytes (approximately 55.95 MiB) | 143,907,468 bytes (approximately 137.24 MiB) |
git ls-files counts tracked entries, potentially including submodules. It does not count files inside submodules or the data a recursive clone would fetch.
Packfile sizes were measured from .git/objects/pack/*.pack after cloning. They exclude TLS, HTTP, and Git protocol overhead, so they are not total network transfer sizes. They also differ from the sizes of the full repository history and the working directory.
Environment and Measurement Method
- OS: macOS
- Git:
git version 2.50.1 (Apple Git-155) - Transport: HTTPS to public repositories; HTTP/2 confirmed through tracing
- Clone options:
--depth=1 --single-branch --no-tags, with a fixed tag - Checkout: Full working-tree checkout using the default clone behavior; no partial clone or sparse checkout
- Submodules: No recursive submodule fetching
- Configuration isolation: Disabled system-level configuration and redirected the global configuration file to
/dev/null - Test configurations: Passed through
git -cwithout modifying configuration files - Measurement interval: From the start to the completion of the clone command, including network transfer, post-receive processing, and checkout
- Timing: Python’s
time.perf_counter() - HTTP tracing:
GIT_TRACE_CURLandGIT_TRACE_CURL_NO_DATA=1 - Iterations: Four configurations tested over three rounds on GitHub and five rounds on GitLab, for 15 and 25 runs respectively. Each run used a fresh directory.
Configurations Tested
| Configuration | Settings |
|---|---|
| Default | Neither setting explicitly specified |
| Compression disabled | core.compression=0 |
| Increased POST buffer | http.postBuffer=1073741824 |
| Combined | Both settings enabled |
GitHub Results
Times are in seconds; all measurements are shown.
| Configuration | Round 1 | Round 2 | Round 3 | Median |
|---|---|---|---|---|
| Default | 8.019 | 8.354 | 8.221 | 8.221 |
| Compression disabled | 8.055 | 36.760 | 8.057 | 8.057 |
| Increased POST buffer | 7.901 | 7.925 | 7.797 | 7.901 |
| Combined | 8.173 | 8.253 | 8.253 | 8.253 |
No consistent improvement was observed. One run with compression disabled took 36.760 seconds; the cause was not investigated.
The following were identical across configurations:
- GET requests: 1
- POST requests: 2
- POST request body
Content-Length: 351 bytes and 201 bytes - Stored packfile size: 58,668,712 bytes
- Packfile SHA-256: Identical across all 15 runs
GitLab Results
All timings are in seconds.
| Configuration | Round 1 | Round 2 | Round 3 | Round 4 | Round 5 | Median |
|---|---|---|---|---|---|---|
| Default | 14.968 | 13.926 | 14.192 | 14.928 | 14.862 | 14.862 |
| Compression disabled | 14.049 | 14.533 | 14.360 | 16.381 | 14.819 | 14.533 |
| Increased POST buffer | 14.432 | 13.681 | 13.955 | 15.094 | 13.455 | 13.955 |
| Combined | 14.704 | 15.420 | 16.587 | 15.634 | 15.147 | 15.420 |
The GitLab results also showed no clear improvement.
The following were identical across configurations:
- GET requests: 1
- POST requests: 2
- POST request body
Content-Length: 375 bytes and 201 bytes - Stored packfile size: 143,907,468 bytes
- Packfile SHA-256: Identical across all 25 runs
Even with a received packfile of about 137 MiB, the POST bodies were below the default 1 MiB threshold.
Conclusion
Client-side settings differ from server-side settings. The same issue often comes up with relational databases.
ci-perf-lint incorporates findings from investigations like this to help speed up CI with a single command.