After facing challenges due to the absence of proper backups, I developed two bash scripts, git-clone-all and git-pull-all, designed to streamline the backup process of GitHub repositories. These scripts facilitate the rapid cloning and updating of all repositories associated with a user or organization, offering customizable settings for both concurrency and commit depth. This approach greatly simplifies the task of maintaining a local copy of all my code, ensuring a smoother transition across different environments.

Introduction

Introduction

We've all experienced that moment of panic when our computer crashes and we realize our backups are outdated. I recently went through this, but the cloud came to my rescue. It was a clear signal that I needed to improve my backup strategy.

Backing up things like photos and music is pretty straightforward since they're often kept in local libraries; your images and tunes are in a folder on your computer or in the cloud.

However, since it's almost always managed with some version control, code tends to be scattered across various places and exists in multiple versions.

Local Code vs Backed-Up Code

Developers keep active code locally and store the rest (along with a copy of their local code) in the cloud, often on sites like GitHub. This enables collaboration and ensures the cloud version is stable while developers can work on yet-unfinished code locally.

What this looks like in practice varies from developer to developer; For me personally, I organize all of my repositories within a single folder named src, with subfolders named after the users or organizations on GitHub whose repositories I clone; those go within the named folders. For example, here's part of the directory listing from my ~/.src/github/f3rno64 folder:

~/.src/github/f3rno64
  ├── http-server-md
  ├── track-time-cli
  └── ...

Just to give you an idea, my folder with all my personal projects for active development is a whopping 8.1GB! 😱 But, when I use git-clone-all, that same folder shrinks down to just 878MB, even though it has way more repositories in it! 📂🔃

This huge difference is mainly because a lot of these repos use Node.JS, and if you've worked with it, you know those node_modules dependency folders can get pretty hefty. 📦💪🏼

Creating A Pristine Code Backup 📂

Manually cloning all of my GitHub repositories for backup seemed tedious. Instead, I decided to invest a bit of time upfront to create a set of versatile bash scripts. Here's what they do:

• Efficiently clone all missing repositories: Perfect for archiving codebases. 📥

• Customizable output: Include full commit histories or just the latest state. 📄

• Handles large-scale cloning: Supports concurrency for cloning hundreds of repositories with ease 🚀

• Updates repositories en-masse: Perfect for keeping your local source tree up to date as your colleagues push changes. 🔄

This scripting approach proved to be a worthwhile investment. Not only did it streamline my backup process, but it offered the flexibility I wouldn't have achieved with manual cloning. 💪

My GitHub Management Toolkit

The results were two new, slim bash scripts that simplified my GitHub workflow, and they're available in my mass-git-scripts repository:

• git-clone-all: clones all repos belonging to a user or organization

• git-pull-all: updates all repos in a directory

These scripts have transformed my workflow! I use them to:

• Rapidly clone all my repositories.

• Quickly onboard to new codebases at work.

• Instantly restore my development environment after a system reset.

For example, to clone all of my repositories, I ran the following (I have 172 repos):

git-clone-all --owner f3rno64 --limit 200 --jobs 30 --dir f3rno64

Here's part of the output, showing the configuration and the actual number of repos found:

This cloned everything (all branches, all tags, complete commit histories) in 58 seconds.

The result was a folder 878MB in size. Not bad when the folder holding the personal repos I am working on is 8.7GB in size.

After cloning, to maintain an updated backup, git-pull-all was needed; I execute it occasionally with a single argument: the path to the folder holding all repositories to be updated:

git-pull-all ./f3rno64

Repositories that have un-commited changes are skipped. In the future, some logic involving git stash would be a nice addition to update even those repos.

---

In Conclusion ✨

With these two scripts in your toolbox, managing your GitHub code becomes effortless! 🛠️ git-clone-all lets you download entire codebases in record time while git-pull-all keeping your local copies in sync with the latest changes. 🔄

What's more, you can use git-pull-all to automate updates for your work projects, eliminating the need to manually pull changes whenever a pull request is merged. Go ahead and try them – your Git workflow will thank you for it! 😊🎉

If you're a developer who relies on GitHub, give these scripts a spin, and let me know what you think! I'm always looking for ways to improve them. Happy coding! 🚀💻

You can find the GitHub repository mass-git-scripts here. Be sure to check out the README!

I'll leave you with the colorized help output of git-clone-all --help: 🎨📃

---

In Conclusion: A Call to Connect

👋 Hey there! Thanks for sticking around till the end of this post. I don’t blog often, but when I do, it’s usually about my personal projects, coding adventures, or helpful tutorials. 🚀

If you’ve found value in what you’ve read here, I’d love to stay connected! Here’s how:

1. Hashnode: Follow me on Hashnode for more tech insights, code snippets, and occasional rants about semicolons. 😉

2. Twitter: Catch my thoughts in 280 characters or less on Twitter. Let’s chat about the latest frameworks, memes, and the eternal tabs vs. spaces debate!

3. Dev.to: Join the community over at Dev.to. I’ll be sharing more in-depth tutorials and diving into the nitty-gritty of my projects.

4. Newsletter: For the VIP treatment, sign up for my newsletter. You’ll get exclusive content, early access to new posts, and maybe even a virtual high-five. 🙌

Remember, tech is all about community, collaboration, and continuous learning. Let’s build cool stuff together! 🤝

Until next time, happy coding! 🎉