My Private, Free AI Setup

This article is republished with permission from Wonder Tools, a newsletter that helps you discover the most useful sites and apps. Subscribe here.

Short on time? Read this 30-second summary of today’s post. 👇

Download a free, private AI program to run on your computer. Use it offline without any subscription cost and avoid the risk of having sensitive info ingested into a large language model like ChatGPT, Claude, or Gemini. The newest versions of private AI tools like Jan run easily on my 2021 Mac laptop, cost nothing, and are easy to use. They’re a good alternative to costlier AI platforms.

🔰 Quick start guide

1. Download and install Jan for free. Other good free alternatives to consider include Msty, AnythingLLM, or LM Studio.
2. Open Jan and pick an open-source large language model. The model you use impacts the AI’s response style. You can switch anytime. I use the v1 model.
3. Try your first query. Here are a few quick mini prompts to start with:
   “Summarize the pros and cons of using AI for [specific task].”
   “Turn my rough notes below into a short summary and bullet points.”
   “Turn this angry email draft to my service provider into a constructive message more likely to generate a helpful response.”
4. Adjust the app’s appearance settings, including font size and shortcuts.
5. Close other processor-intensive apps on your computer, like video editing tools, to reduce the likelihood of your computer slowing down.

🕵🏻 5 reasons to use private AI

• Save money: Avoid subscription fees by running AI models on your own computer. Generate unlimited responses without monthly charges.
• Keep your data private: Using private AI on your computer ensures no data is sent to or stored on big tech firms’ servers. No conversations leave your device. You can even run these tools offline.
  • For sensitive legal, medical, financial or personal issues, ask questions without worrying about your data ending up in a large language model’s training data.
• Work offline: Having full offline access is handy whether you’re traveling without Wi-Fi, working in a remote area, or hesitant to trust a random public network.
• Experiment with hundreds of open-source models: Choose an open-source large language model that suits you. Each is trained differently. Some are stronger at certain languages, others specialize in coding. New ones emerge regularly. Switch as often as you’d like. By contrast, ChatGPT, Claude, Copilot, and Gemini limit you to the platform’s own models.
  • Tip: Use LM Arena to compare two models’ responses side by side.
• Reduce your environmental impact: If you run hundreds of daily prompts, a local AI app may mean less use of internet infrastructure and remote data servers.

💫 Jan is an excellent, free, private AI tool

• Platforms: Mac, PC, Linux

What I like about it

• Fast and easy to set up and use: Jan takes a minute to download and install. Using Jan is as easy as using ChatGPT, Claude, or any other chatbot, though you do have to make an initial decision about which model to use.
• Assistants: Create customized AI helpers for various purposes. One for translating Chinese, another for coding. Task it to “Act as a software engineering mentor focused on Python and JavaScript. Provide detailed explanations with code examples. Use markdown formatting for code blocks.”
• Projects: Organize queries into distinct folders for easy access to subjects of interest without searching through hundreds of threads.
• Integrations: Link Jan to Canva, Todoist, Linear, or other tools using MCP (model context protocol) connections.
• Documentation and resources: Lots of useful documentation, including a handbook and blog.

What’s Next: Jan AI is developing mobile versions for iOS and Android and adding integrations to link Jan to other services.

🩺 A Jan case study

Becki Lee, a senior technical writer, uses Jan to explore health questions she wants to keep private.

“I have a chronic illness I’m struggling to get diagnosed,” she emailed me. “So I created an assistant to help interpret test results and brainstorm possible explanations for my symptoms. Obviously, it’s super important to take this with a grain of salt (a chatbot is absolutely no substitute for a doctor). However, this helps bubble up conditions I can research further on my own, and it also generates questions I can ask my actual doctor.”

✨ More free AI options for Mac, PC, or Linux

Msty

The free version of this well-designed app has multiple unique features. Unlike Jan, which is completely free, Msty also has paid advanced features.

Its best free features include:

• A built-in prompt library with hundreds of options.
• Special focus and zen modes that strip away side menus.
• Create multiple personas, which are assistants with distinct personalities. Each can adopt a different style or approach in answering your queries.
• Knowledge Stacks let you import document collections for analysis. These can include PDFs, Word documents, PowerPoints, spreadsheets, lists of YouTube links, or even an Obsidian vault.
• Advanced features, like multistep automations, require a paid subscription. I’ve only used the free version. It’s easy to use, powerful, and well designed. I chose the Gemma 3.

AnythingLLM

Like Jan, this is a straightforward open-source AI app that’s a good option for novice AI users.

How it’s different from Jan

• You can upload files for AnythingLLM to summarize.
• Enable it to make simple charts.
• Turn on Web search, which requires a free API key from Google or Serpa.
• There’s also a new beta Android version.

Caveat: It’s not quite as nicely designed as Jan, and isn’t updated as often.

LM Studio

This more developer-friendly option is less simple for beginners.

What’s notable: Florent Daudens, an AI expert and educator who used to oversee daily editorial coverage at CBC/Radio-Canada, relies on LM Studio for private AI use.

I asked him why and he said, “It’s practical, with a user/developer-friendly interface, quick updates when new models drop, a server option, and helpful model compatibility info.”

In a LinkedIn post, Florent shared an example of using LM Studio on his laptop. He used Google’s Gemma 3 model to analyze plane photos for extracting registration numbers as an investigative journalist might, without sending data to external servers.

Limitations of private AI tools

• Feature limits: Many special features on other AI platforms won’t work on these private AI platforms. ChatGPT’s new plug-ins for Canva or Figma, for instance, won’t work with private AI. You may not be able to export results directly to Google Sheets or Slack, as you can with other AI tools.
• No interactives or advanced visuals: You can’t create infographics and visual illustrations like ChatGPT’s. No coding and hosting interactive applications, as you can with Claude or Gemini. No advanced searches with detailed citations like those from Perplexity.
• Quality variation: Some open-source models have limited or older training data, so results for certain queries may be worse. For ordinary queries and text summarization, this quality difference may not be noticeable.
• Slower speed: Depending on your query, you might wait longer with some open-source models than with ChatGPT, Copilot, or other private AI platforms. Speed hasn’t been a big concern for me so far.
• Can’t handle as much text at once: A smaller “context window” means that private AI tools may not be able to analyze text blocks as large as those ChatGPT or Claude can handle. Some small language models may resort to skimming longer text. They may also be more likely to hallucinate details if asked for summaries of long, complex documents.

🧑‍🎓 Additional resources

• Free, open-source AI tools for journalists curated on Hugging Face by Florent Daudens. Read more about why I like Hugging Face as an open-source AI hub.
• Local LLM Group on Reddit, with 546,000 members. Keep up on notable research on AI and private AI tool development.
• Helpful write-up about local large language models by Stephen Turner.
• LinkedIn Learning Course on private large language models and Jan AI.

This article is republished with permission from Wonder Tools, a newsletter that helps you discover the most useful sites and apps. Subscribe here.

View the full article