Step-by-Step Tutorial: Building Your Local AI System

Prerequisites

• Operating System: Linux, macOS, or Windows.
• Python 3.8+ and pip: For managing dependencies.
• Node.js and npm: Required for many MCP servers, including the filesystem server we'll use. Download the LTS version from nodejs.org.
• Git: For cloning repositories.
• C/C++ Compiler: For building `llama.cpp` (e.g., GCC on Linux, Xcode Command Line Tools on macOS, Visual Studio on Windows).

Step 1: Set Up Your Development Environment

Download and install Node.js from the official website: nodejs.org. This will also install npm.

Verify installation:

node --version
npm --version

Check if Git is installed: `git --version`.

If not, install it via your system's package manager (e.g., `sudo apt install git` on Ubuntu, `brew install git` on macOS).

Step 2: Run Qwen3.6 Locally with `llama.cpp`

Open your terminal and run:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make -j # Use -j to utilize all CPU cores for faster compilation

If you have an NVIDIA GPU, you'll want to enable CUDA support for faster inference. On Linux, this typically looks like:

git pull
make clean
make -j LLAMA_CUBLAS=1

For Apple Silicon (Metal), GPU support is usually enabled by default. For Windows, building with CUDA might involve Visual Studio; refer to the official llama.cpp GitHub repository for detailed instructions.

Qwen3.6 models are available in GGUF format on Hugging Face. We'll use a quantized version for better performance on consumer hardware. For this tutorial, let's pick Qwen3.6-27B. You can find various quantized versions (e.g., Q4_K_M) on the Unsloth Qwen3.6-27B-MTP-GGUF Hugging Face page.

Download your chosen GGUF file (e.g., `Qwen3.6-27B-MTP-UD-Q4_K_M.gguf`) to a directory like `~/models/qwen3.6/`.

# Example: Create a directory and navigate into it
mkdir -p ~/models/qwen3.6
cd ~/models/qwen3.6

# Download the model (replace with your chosen model URL)
# Example using curl (you might need to copy the direct download link from Hugging Face)
curl -L -o Qwen3.6-27B-MTP-UD-Q4_K_M.gguf https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF/resolve/main/Qwen3.6-27B-MTP-UD-Q4_K_M.gguf

Navigate back to your `llama.cpp` directory. Start the `llama.cpp` server, pointing it to your downloaded Qwen3.6 model. This server will expose an API that MCP clients can easily connect to.

cd ~/llama.cpp
./server -m ~/models/qwen3.6/Qwen3.6-27B-MTP-UD-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ngl 999

• `./server`: Starts the `llama.cpp` API server.
• `-m ~/models/qwen3.6/Qwen3.6-27B-MTP-UD-Q4_K_M.gguf`: Specifies the path to your Qwen3.6 GGUF model.
• `--host 0.0.0.0`: Makes the server accessible from other devices on your local network. For local-only access, use `127.0.0.1`.
• `--port 8080`: Sets the port for the API server.
• `-ngl 999`: Offloads as many layers as possible to the GPU (if `LLAMA_CUBLAS=1` was used during compilation). Adjust `999` to a suitable number if you have less VRAM or want to fine-tune layer distribution.

Keep this terminal window open; the server needs to be running for your AI system to work. You should see output indicating the server has started and is listening on port 8080.

Step 3: Install and Configure MCP Servers

Open a new terminal window (keep the `llama.cpp` server running). The Filesystem MCP server allows your AI model to read, write, and manage files on your local machine securely.

npm install -g @modelcontextprotocol/server-filesystem

MCP servers often require configuration to define what resources they can access. For the Filesystem server, you'll typically specify allowed directories to ensure security and prevent the AI from accessing arbitrary parts of your system. Create a `server_config.json` file in a convenient location (e.g., in a new `mcp-servers` directory).

mkdir -p ~/mcp-servers
cd ~/mcp-servers
nano server_config.json

Add the following content, replacing `"/path/to/your/workspace"` with the actual path to a directory you want your AI to be able to access (e.g., `"/home/youruser/my_ai_projects"`):

{
  "allowedPaths": [
    "/path/to/your/workspace"
  ]
}

Save and close the file.

From the `~/mcp-servers` directory, start the server:

npx @modelcontextprotocol/server-filesystem --config server_config.json --port 8081

• `--config server_config.json`: Points to your configuration file.
• `--port 8081`: Runs the MCP server on a different port than your LLM server to avoid conflicts.

You should see output indicating the Filesystem MCP server has started and is listening on port 8081. Keep this terminal window open as well.

Explore Other MCP Servers: The MCP ecosystem is growing rapidly. You can find servers for GitHub integration, web search, database access, and more. Check out the awesome-mcp-servers GitHub repository for a comprehensive list.

Step 4: Connect an MCP-Compatible Client to Your Local Qwen3.6

In your chosen client's settings, look for an option to configure the LLM provider. You'll typically set:

• API Base URL: `http://127.0.0.1:8080/v1` (or `http://localhost:8080/v1`)
• API Key: Usually "none" or an arbitrary string, as `llama.cpp`'s OpenAI-compatible server doesn't require one by default.
• Model Name: The name of your Qwen3.6 model (e.g., `Qwen3.6-27B-MTP-UD-Q4_K_M`).

Clients that support MCP will have a section to add or discover MCP servers. You'll need to provide the URL of your running Filesystem MCP server.

Add a new MCP server with:

• Display Name: `Local Filesystem`
• URL: `http://127.0.0.1:8081` (or `http://localhost:8081`)
• Authentication: Often "none" for local servers, or follow specific instructions if the server requires it.

Once configured, the client will connect to both your local Qwen3.6 instance and the Filesystem MCP server. The model will then be able to "discover" the tools exposed by the Filesystem server.

Step 5: Define and Call Tools via MCP