DeepFellow Infra runs models on your hardware: on-premise, in a private cloud or hybrid. It also connects cloud providers if you allow them, so one layer covers every model your organization uses. Only Server can talk to Infra; employees never connect to it directly.
Load balancing
Install the same model several times: on more GPUs, on more machines or on other nodes of the mesh. Infra decides where each request goes.
- Warm first. A request goes to an instance where the model is already loaded, then to the one with more capacity. vLLM and SGLang report their real capacity, sized from the memory they reserved.
- Fill, then spill over. Each instance takes requests up to its capacity before the next one gets any, so models do not sit loaded on more GPUs than the traffic needs.
- Live load across machines. Instances on other nodes count as full at 90% of their capacity, to cover the delay in load updates.
- No refusals at the peak. When every instance is full, the request goes to the least loaded one instead of failing. A cloud provider serving the same model takes the overflow only after your own warm instances.
Mesh: many machines, one Infra
Connect Infra instances into a tree. Server talks to one node and reaches every model in the tree.
- Secure joining. Each node joins one parent with a mesh key, and both sides verify each other. A node with no mesh key set accepts no one.
- Everything is shared. Models, MCP tool servers and custom services of every node are available through its parent. Child nodes can use their ancestors' models; you can switch that off.
- Live state. Load, loaded models and tool health travel over the mesh as they change.
- Visible topology. The admin panel and API show the tree from each node's point of view.
- Grow without changes. Add a GPU server in another location, a Mac for smaller models or a node with document services. Applications keep the same address.
