An open source inference server brings continuous batching and SSD tiered caching to Apple Silicon, so one Mac can answer concurrent AI requests for a whole team instead of a rented GPU that bills by the hour.
The support assistant that drafts replies, the internal doc search your team hits all morning, the classifier grinding through inbound leads: the second two of those need an answer at the same moment, you no longer have a local model problem, you have a server problem. The standard fix is renting a GPU and keeping it warm, and one A100 at Lambda's published on-demand rate of $2.79 an hour comes to roughly $2,000 across a month you never switch off. jundot/omlx, which climbed GitHub's daily trending list today, argues that a Mac you buy once, starting at $1,999, can cover the same shift.
Plenty of tools already run a model on hardware you own. Ollama made that a one command affair a while ago. But the thing separating a local runtime from an actual server is what happens on the second simultaneous request, and that gap is precisely what omlx is built to close.
It does continuous batching, which means it interleaves concurrent requests through the model instead of making them queue. That one property is the difference between a model a single person is chatting with and a model twelve people plus three background jobs are hitting at once.
The second piece is a tiered KV cache. Frequently used blocks stay in RAM, colder ones get pushed out to SSD, with prefix sharing and copy on write between the tiers. In practice that means the long system prompt and the 40 page policy document your assistant keeps rereading get processed once and reused, and the cache survives even when the conversation changes direction mid thread. Anyone who has watched a per-token bill balloon on repeated context will recognize what that is worth.
Coverage is wider than chat, too. It serves text models through mlx-lm, vision language models like Qwen3.5 and Pixtral, OCR models including DeepSeek-OCR, plus embedding models and rerankers. That is most of what a retrieval stack actually needs, in one process you manage from the macOS menu bar. On speed, the project reports that its native kernels for GLM-5.2 make prefill roughly 30 times faster, 845 tokens per second against about 29 with the generic fallback. Read that number carefully though: it was measured on an M3 Ultra, which sits a long way up the price list from the $1,999 machine. The repo itself clears the production readiness bar, with 18,916 stars and 1,642 forks since February, Apache 2.0 licensing, commits landing today, and installation through a disk image or a Homebrew tap rather than a pile of manual steps.
So who should actually care? The ops lead at a 30 person company who wants an internal assistant over company documents without routing anything through a third party. The agency running document extraction for clients where per-page OCR pricing eats the margin. The regulated shop in healthcare, legal, or finance where "the data does not leave the building" is a contract term rather than a preference. And the engineering manager who already has Macs in the office and would rather spend $2,000 once than $2,000 every month, forever, on capacity that sits idle all night.
Now the honest part. This is Apple Silicon and macOS 15 or newer, full stop, so if you run a Windows or Linux shop there is no conversation to have. The default memory ceiling is system RAM minus 8GB, which on a base 36GB Mac Studio leaves under 30GB for weights and cache, and that ceiling decides which models you can load at all. The ones you want may not fit. Getting those fast native kernels requires a full Xcode install rather than just the command line tools. Multi Mac distributed inference exists but the project labels it experimental, so treat it that way. A Mac in a closet also has no SLA, no failover, and nobody paged at 3am except you, which is tolerable for an internal tool and reckless for anything customer facing. A rented A100 will still beat it on raw throughput, as well. The claim worth defending is sufficiency for internal load, not parity.
The number that matters here was never tokens per second. It is that the capacity most companies rent by the hour, and keep paying for while it sleeps, now fits on a desk for about what one month of the invoice used to cost.