Presenting FileFusion: https://github.com/drgsn/filefusion - a powerful file concatenation tool designed specifically for Large Language Model.

The Problem: Context Management for LLMs

Let's face it: working with LLMs is like trying to explain your code to a brilliant but extremely literal-minded colleague who hasn't had their morning coffee yet. They need context. Lots of context. And while IDE plugins are great, they sometimes fall short when you need comprehensive codebase understanding.

I found that using the project features in Claude and ChatGPT – where you can upload chunks of codebases – often yielded the best results. But there was a catch (isn't there always?): managing what code goes into your LLM context is crucial for getting meaningful responses, and keep it synced with the changes from git is a pain.

That's where FileFusion comes in, like a helpful barista serving up perfectly filtered code to our caffeine-deprived LLM friend.

How LLMs "Understand" Code

Before diving into FileFusion's features, let's talk about how LLMs process code. You might think they read code like we do, but they're more like that friend who memorized the entire dictionary but still occasionally uses words in hilariously wrong contexts.

LLMs don't really "understand" code in the way human programmers do. Instead, they:

1. Process Code as Tokens: When you feed code to an LLM, it breaks it down into tokens - small chunks of text. For example, the line function calculateTotal() might be broken into tokens like ["function", "calculate", "Total", "(", ")"].

2. Look for Patterns: LLMs have been trained on massive amounts of code, so they recognize patterns. They're like that senior developer who's seen so many codebases they can spot a design pattern from a mile away – except the LLM has seen millions of codebases.

3. Use Statistical Relationships: The model understands relationships between different parts of code based on statistical patterns in its training data. It's similar to how you might guess what a function does based on its name and context, except the LLM is doing this based on probabilities learned from its training data.

4. Context is Everything: LLMs rely heavily on context to generate meaningful responses. The more relevant context they have, the better they can understand and respond to questions about the code. It's like trying to debug a function – the more context you have about where and how it's used, the better you can understand what it's supposed to do.

This is why cleaning and optimizing code for LLM consumption is so important. Comments might be helpful for humans, but they can actually confuse LLMs by adding noise to the statistical patterns they're looking for. Similarly, repetitive boilerplate code like getters and setters can dilute the meaningful patterns in the code.

Smart Pattern Matching with Glob Patterns

Now, let's talk about how FileFusion helps you select exactly what code to feed your LLM friend. It uses glob patterns, which are like regular expressions' more approachable cousin. You know, the one that doesn't make you question your life choices every time you use it.

# Process only Python and JavaScript files
filefusion --pattern "*.py,*.js" /path/to/project

# Exclude test files and vendor directories
filefusion -e "**/*_test.go,vendor/**" /path/to/project

# Complex pattern matching (for when you're feeling adventurous)
filefusion -p "src/**/{test,main}/*.{js,ts}" -e "**/node_modules/**"

The pattern matching is powered by the doublestar library, which is like filepath.Match after it went to the gym and got some serious upgrades:

• ** for recursive directory matching (because life's too short to list every subdirectory)

• Brace expansion for alternatives, because you don’t want to type the same pattern with different extensions

• Multiple patterns support (because sometimes one pattern just isn't enough)

• Negative patterns with ! (for when you need to tell your build artifacts to kindly see themselves out)

The Cleaning Engine: Tree-sitter Integration

Now for the really cool part: FileFusion uses tree-sitter to perform language-aware code optimization. If you're not familiar with tree-sitter, think of it as a really smart code parser that turns your code into a structured tree faster than you can say "abstract syntax tree" three times fast.

Here's how FileFusion puts tree-sitter to work:

1. Language Detection: First, we figure out what language we're dealing with (because trying to parse Python like it's Java would be... interesting):

func (p *FileProcessor) detectLanguage(path string) cleaner.Language {
    ext := strings.ToLower(filepath.Ext(path))
    switch ext {
    case ".go":
        return cleaner.LangGo
    case ".java":
        return cleaner.LangJava
    // ... more languages (because we're inclusive like that)
    }
    return ""
}

2. Syntax Tree Analysis: The code gets parsed into a tree structure, kind of like a family tree for your code, but with less dramatic holiday dinners:

tree := parser.Parse(nil, input)
root := tree.RootNode()

// Now we can traverse the tree and identify specific nodes
if node.Type() == "comment" {
    // Handle comments (or in some cases, show them the door)
}

3. Smart Cleaning: The cleaning engine can identify and remove:

   • Comments (while preserving documentation if desired)

   • Getter/setter methods (because even LLMs don't need to see getX() returning x for the millionth time)

   • Logging statements (sorry, debug logs, it's not you, it's us)

   • Import statements (when they're not needed)

   • Empty lines and excess whitespace

Concurrent Processing for Performance

FileFusion processes files concurrently using Go's goroutines and channels, because I wanted to use goroutines. I’ve insisted on making my life hard, even though sequencial processing would’ve been probably fast enough:

func (p *FileProcessor) ProcessFiles(paths []string) ([]FileContent, error) {
    numWorkers := min(len(paths), 10)  // Because 11 would be showing off
    results := make(chan FileResult, len(paths))
    jobs := make(chan string, len(paths))

    // Start our worker pool (like a pool party, but for goroutines)
    var wg sync.WaitGroup
    for i := 0; i < numWorkers; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for path := range jobs {
                result := p.processFile(path)
                results <- result
            }
        }()
    }

    // The rest of the concurrent processing magic...
}

Learning Go: Challenges and Insights

Building FileFusion was an excellent way to learn Go, though at times it felt like trying to learn juggling by starting with chainsaws. Some of the "fun" challenges included:

1. Goroutine Management: Ensuring proper synchronization and preventing goroutine leaks. It turns out "let them all run wild" is not a valid concurrent programming strategy.

2. Error Handling: Getting used to Go's error handling patterns. Yes, you have to check every error. No, crossing your fingers isn't an acceptable error handling strategy.

3. File System Operations: Working with filesystem operations concurrently required careful handling of race conditions. Because having multiple goroutines trying to access the same file is like having multiple people try to go through a revolving door at once – chaos ensues.

4. Memory Management: Processing large files efficiently meant being mindful of memory usage. But if your code files are that big, then that’s on you, i don’t think i’ll optimize for it.

Conclusion

FileFusion started as a "simple" learning project and turned into a full-fledged tool that makes preparing code for LLMs a breeze. While there were certainly moments where I questioned my life choices (particularly during concurrent file processing debugging sessions), the end result is something I'm proud of.

The tool fills a specific niche in the LLM workflow, helping developers prepare their code for LLM consumption by cleaning out unnecessary elements while preserving essential structure.

Future enhancements might additional cleaning options, and perhaps integration with popular LLM platforms to automate the process of creating and updating projects. The project remains open source and welcomes contributions from the community. Just please don't submit pull requests written fully by an LLM without proper testing.

The Browser Wishlist: More Than Just Pretty Tabs

Before diving into alternatives, I sat down and really thought about what matters in a modern browser:

• Resource efficiency, I want to be able to browse the internet for a long time.

• Vertical tabs that actually make sense (not just tacked on as an afterthought)

• Rock-solid security and privacy features

• Support for my essential extensions, my carefully curated collection of tools.

• Smart workspace management because my brain works better with organized chaos

• Native-feeling performance

Sounds simple enough, right? Well, buckle up – this journey got interesting fast.

Zen Browser: The Beautiful But Complicated Date

First up was Zen Browser, which caught my eye with its stunning Firefox-based interface. It's like someone took Firefox to a design workshop and came back with a masterpiece.

What made me swoon:

• Those gorgeous design and vertical tabs that look stunning.

• Workspaces that actually understand how people work

• A split-view system that makes research feel natural

• "Zen Glance" preview feature (goodbye, risky clicks!)

• Powerful theming through "Zen Mods"

• Firefox's battle-tested privacy features

• A treasure trove of Firefox extensions

But then reality hit. No DRM support meant saying goodbye to some streaming services, and Google Meet (a daily necessity) performed like it was running on a potato. It's like dating someone who looks perfect on paper but snores like a chainsaw – some deal-breakers you just can't ignore.

Vivaldi: The Overly Customizable Ex

Remember that friend who got really into modding their car and now can't stop talking about torque ratios? That's Vivaldi. It's incredibly powerful, but sometimes power isn't everything.

I tried the custom CSS route with themes like VivalArc and Vivaldi VH. The results looked amazing, but maintaining custom CSS felt like adopting a high-maintenance pet – one that needs attention after every single update. It's the browser equivalent of dating someone who needs you to compliment their outfit five times before leaving the house.

Don’t get me wrong, the browser is great, it feature packed, and it was my main browser for a week. I even created homepage.drgos.com to be my homepage for it. But i don’t want to tinker with it that often, that what i was feeling like, everyday making a small change.

Dear Vivaldi team: How about letting users share complete configurations? Reducing the barrier to entry for new users and letting existing users be your megaphone. Imagine clicking once to import someone's perfect setup instead of spending hours in settings menus. Do some competitions on X.com or Reddit and give some prices for the ones with the most votes. Just a thought!

SigmaOS: The Productivity Guru Next Door

Walking into SigmaOS feels like entering a minimalist's dream office. Everything has its place, and there's a place for everything. It's not just another browser; it's a complete reimagining of how we should interact with the web.

The standout features read like a productivity wishlist:

• Workspaces that feel like well-organized rooms in a digital house

• Airis AI assistant that i haven’t used, so i can’t say anything about it.

• Breadcrumb navigation that maps your diving expeditions into Wikipedia

• Universal "Lazy Search" that finds anything, anywhere (because who has time to remember where they saw that thing?)

• Dynamic theming that plays nice with every website

• Built-in ad blocking that doesn't break sites

• Team collaboration tools that don't feel bolted on

Using SigmaOS feels like having a personal assistant who organizes your digital life while you sleep. It's perfect for those who color-code their calendar and alphabetize their spice rack.

Orion Browser: The Unexpected Perfect Match

Sometimes the best things in life come from unexpected places. Enter Orion Browser, I discoverd it through a youtube video recommandation. It’s built on WebKit, similar to SigmaOS. It's like finding out that quiet person at work is actually the most interesting one there.

What makes Orion special:

• Speed - the browser feels really snappy and fast.

• The holy grail of extension support Chrome AND Firefox. But some extensions might not work as expected. Luckly the 4 extentions that i use are compatible.

• Vertical tabs that feel natural, not forced with breadcrumbs

• Mac integration that feels like it came from Apple itself especially when in compact mode.

• Transform websited into native apps.

• Picture in picture when you want to watch youtube while working on that document.

• Privacy features or at least they advertise that.

• Dynamic theming - it’s more of a miss for me, but it’s not a deal breaker. SigmaOS dynamic theming is better.

Even the Google Meet issue (no virtual backgrounds) had a simple fix using macOS features. It's these thoughtful solutions that make Orion feel like home.
Also i like a lot more the way share screen / tab looks on webkit than on Chromium.

A Surprising Discovery: The Search Engine Renaissance

During this browser adventure, I stumbled into another rabbit hole – the world of alternative search engines. Turns out, while we've all been googling away, a whole ecosystem of innovative search tools has evolved.

There's Kagi (made by the Orion folks) with its privacy-first approach, You.com with its AI-powered results, Perplexity for those who want ChatGPT-style answers to everything, Phind for developers who need technical answers fast, and even Brave Search building its own independent index and a lot more. I know that everyone knows DuckDuckGo, but there are a lot more search engines out there.

Each one brings something unique to the table. I’m not the biggest fan of Perplexity and You.com as main search engine, even though i use them from time to time. Especially when i want to buy some product and I want to compare multiple options.

Kagi caught my attention with its promise of unfiltered, ad-free results, so I'm giving it a trial run. Who knows? Maybe finding the right search engine is just as important as finding the right browser.

The Happy Ending

After weeks of testing and probably confusing every website's tracking algorithms, I've settled into Orion Browser. It hits that sweet spot between innovation and stability, power and simplicity. It's like finding the perfect coffee shop – not too busy, not too quiet, with just the right amount of character.

I will keep SigmaOS installed so that i can give Airis AI a try.

Key Takeaways

• The perfect browser isn't about having every feature – it's about having the right features

• Beautiful design needs to be backed by solid performance

• Sometimes the best solution comes from the most unexpected place

• Privacy and functionality aren't mutually exclusive

• The browser market is more exciting than it's been in years. For a while i thought that everything will go Chromium, I’m exited to see that Firefox and WebKit hang in there.

Words of Wisdom

Finding your ideal browser is like finding your favorite pair of jeans – what works perfectly for one person might be completely wrong for another. Don't be afraid to experiment, and remember that it's okay to change your mind as your needs evolve.

P.S. Remember, the "perfect" browser today might not be perfect tomorrow. Technology evolves, our needs change, and that's okay. The joy is in the journey of discovery – and having a reliable browser to help you along the way!

A commit, in its essence, is a snapshot of your project at a specific point in time, encapsulating all your changes and updates. It's more than just a mere save point.

The Significance of a Commit

• Building Blocks of Your Project: Commits are the building blocks of your development work. They represent incremental steps towards your project's completion, each with its own unique identity.

• Facilitating Collaboration: In a team environment, commits are pivotal. They allow multiple developers to contribute to the same project without overwriting each other's work. By committing changes and merging them, teams can seamlessly integrate their efforts.

• Rollback and Error Correction: One of the most powerful features of commits is the ability to roll back to a previous state. If a new code introduces a bug, you can revert to a previous commit, ensuring stability and reducing downtime.

• Branching and Merging: Commits are integral to branching and merging strategies in Git. They allow you to diverge from the main line of development, experiment, and then bring those changes back into the main project safely.

Decoding the Components of a Git Commit: The Building Blocks of Project History

When you create a commit in Git, you're doing much more than just saving your work. You're creating a comprehensive package that records the state of your project at a particular moment. Understanding the components of a commit is essential for leveraging Git's full potential. Let's break down these components to appreciate the detailed story each commit tells.

Core Components of a Git Commit

1. Snapshot of Changes

   • Files Changed: This includes modifications to existing files. Git tracks the differences (or 'diffs') from the previous commit, allowing you to see exactly what was altered.

   • Files Added: New files that you've created and staged for the commit are listed here. They become part of the project's tracked files from this commit onwards.

   • Files Removed: If you've deleted files from your project, this removal is also captured. These files are no longer tracked by Git after the commit.

2. Commit Message

   • Essential for Communication: The commit message is where you describe what changes were made and, more importantly, why. Good commit messages are crucial for communicating with team members and your future self.

   • Structure: A good commit message typically has a short, descriptive title, followed by a detailed body explaining the context and rationale for the changes.

3. Unique Identifier (SHA-1 Hash)

   Fingerprint of Your Commit: Every commit is assigned a unique SHA-1 hash, a 40-character string that acts as the commit's fingerprint. This identifier is crucial for Git's version control mechanism.

4. Author and Committer Information

   Who Made the Changes: Git records the author's name and email address with each commit. This is typically the person who originally wrote the changes.

5. Parent Commit Reference

   • Linking the Project History: Every commit (except the initial one) in Git has a parent commit. This link forms a chain that Git uses to track the entire history of the project.

   • Branching and Merging: Understanding parent commits is key when dealing with branches and merges, as it helps Git to correctly combine different streams of work.

The Essence of a Good Commit

The Ideal Size and Scope of a Commit

• Atomicity: A commit should represent a single logical change. Regular commits after small changes are easier to track and manage.

• Small Size: Smaller commits are preferable for ease of understanding and rollback. Group related changes into a single commit.

• Testing Before Committing: Ensure that the new code does not break existing functionality.

• Cleanup code: remove commented code, fix TODOs, remove unused imports, fix linting.

• Avoid Temporal Commits: Avoid commits like "End of day commit" or "Fixing typo.". You can use these as checkpoints while iterating on the implementation, but don't push them to remote. Soft reset the changes and write comprehensive commits and changes are final.

• Example: Adding a function is a good commit size; overhauling an entire module is not.

Writing Good Commit Messages

• Start with a concise subject line: Summarize the change in 50 characters or less.

• Add a detailed description (optional): Explain the 'what' and 'why', not the 'how'.

• Use the imperative mood: "Fix bug" rather than "Fixed bug" or "Fixes bug".

Examples of Commit Messages

Good Commit Messages

1. Descriptive and Concise Title with Detailed Body:

   • Title: "ISSUE_ID: Implement user authentication in login module"

   • Body:

     • "Added JWT-based authentication mechanism."

     • "Included error handling for failed login attempts."

     • "Updated unit tests to cover new functionality."

2. Clear and Specific, Explaining the 'Why':

   • Title: "ISSUE_ID: Fix memory leak in image processing function"

   • Body:

     • "Identified and resolved a memory leak issue where buffers weren't being freed."

     • "Refactored buffer management to improve performance and reliability."

     • "Added comments for clarity on buffer handling logic."

3. Use of Imperative Mood, Reflecting Action Taken:

   • Title: "ISSUE_ID: Refactor database query functions for scalability"

   • Body:

     • "Revised SQL queries to use prepared statements, enhancing security and performance."

     • "Optimized query logic to reduce execution time for large datasets."

     • "Updated relevant documentation to reflect changes in query functions."

Most git providers provide integration with issue-tracking software. So it's a good idea to include the id of the task / issue for easy navigation between the issue and commit an vice-versa.

Bad Commit Messages

1. Vague and Non-Informative:

   • "Fixed stuff"

   • "Updates"

   • "Made changes"

2. Too Broad, Lacking Specificity:

   • "Worked on app"

   • "Changes in code"

   • "Bug fixes"

3. Personal Messages Not Useful for the Team:

   • "Trying something out"

   • "Not sure if this will work"

   • "Committing so I can leave for the day"

4. No Explanation of 'Why':

   • "Changed login"

   • "Edited queries"

   • "Updated files"

You can have temporary bad commit messages while you experiment but don't forget to soft reset the changes and replace the bad commits with good ones.

 git reset --soft $(git merge-base your_main_branch HEAD)

This will move the git history back to the head of your_main_branch and will allow you to push commit over that on your branch.

Key Takeaways for Commit Messages

• Good Commit Messages: Are specific, informative, and explain not only what was changed but why. They follow a clear structure with a concise summary and an optional detailed description.

• Bad Commit Messages: Are vague, overly broad, or too personal. They lack the necessary information to understand the context and rationale behind the changes.

• A good commit is characterized foremost by its size; it should be small and focused, encompassing a single logical change. This granularity not only simplifies understanding and review but also eases the process of troubleshooting and reverting changes if needed. The changes within each commit should be logically grouped, ensuring that all modifications contribute cohesively to a specific functionality or fix. Before committing, it's imperative that the changes are thoroughly tested to maintain the stability and reliability of the codebase. Equally important is the cleanliness of the code in a good commit. It should be free from unused imports, commented-out code, and unresolved 'TODOs'

The power of the terminal lies in its flexibility and the ability to customize it to fit our unique workflows. Drawing inspiration from Git hooks, a powerful feature in Git that triggers custom scripts at certain points in its execution flow, we can similarly extend the functionality of our terminal. This article delves into how I used this concept to streamline my workflow, not just limited to Go development, but applicable to a range of tasks in the terminal.

Understanding Git Hooks

Git hooks are scripts that Git executes before or after events such as commit, push, and pull. They are a vital part of automating tasks in the software development process. This concept can be extrapolated to the terminal, where we often perform repetitive tasks that could benefit from automation. For example, I automated the prefixing of each of my commit messages with the ticket id and checks that I don't commit any secrets in my repos (not that well optimized).

Identifying Repetitive Tasks

My journey began with recognizing a repetitive task: running asdf reshim golang every time after using go install or go get. This was necessary due to using asdf as a version manager for Go, but it was easy to forget. This scenario is just one example; think about your daily terminal use and identify similar repetitive commands.

Implementing the hook

To automate this, I created a bash script named .go_install_hook.sh:

#!/bin/bash
# Display a custom pre-run message.
echo "Running a Go command..."

# Run the original 'go run' command.
go "$@"

# Check if the command is 'go run' or 'go install'
if [[ "$1" == "run" ]] || [[ "$1" == "install" ]]; then
  # If it's 'go run' or 'go install', run 'asdf reshim golang'
  echo "it's 'go run' or 'go install', running 'asdf reshim golang'"
  asdf reshim golang
fi

This script acts as a wrapper around your intended command, in my case go whatever allowing you to execute additional logic before or after the main command.

In the example above I print a statement, this is more for debugging purposes, after that I run the intended go command and at the end I check if the command was go run or go install and run the reshim command.

Integrating with the Terminal

The script was then integrated into my daily workflow by aliasing the original command in my bash configuration (.zshrc in my case):

alias go="~/.go_install_hook.sh"

In my case, it was the go command, but this can be adapted to any command where you see a pattern of repetitive post-command actions.

Expanding Beyond Go

While my initial use case was specific to Go and asdf, the principle applies universally. For instance, you could create a script that cleans up temporary files after compiling a program, or one that automatically pushes changes to a remote repository after a successful commit.

Conclusion: A World of Possibilities

This approach opens up a world of possibilities in terminal automation. By emulating the functionality of Git hooks, we can create a more efficient, error-resistant, and customized command-line experience. This method's adaptability across various commands and workflows demonstrates the power of custom scripting in enhancing our daily interactions with the terminal.