X Algorithm

Open source alternative to BytePlus Recommend and Shaped

34K stars
Apache-2.0
Active this week
Last commit 1 day ago

Last verified Nov 1, 2026

Build recommendation systems using X's open-sourced For You feed algorithm, with transformer ranking, visibility filtering, and ML-based retrieval in Rust.

Visit website GitHub34K

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who X Algorithm is for#

ML engineers building content ranking systems

The Phoenix model and training code let you study a production transformer-based ranker, understand how action probabilities combine into a single score, and adapt the scoring weights for your own engagement signals. The synthetic data generation makes a proof-of-concept training run possible without production data.

Skip if:

You need a managed recommendation API that handles serving without infrastructure work. X Algorithm requires building and operating the full pipeline yourself; if the goal is a working recommendation API in days, BytePlus Recommend or Shaped handles that without engineering overhead.

Platform engineers designing feed architecture

The multi-stage request path (query hydration, candidate sources, hydration, filtering, scoring, selection, post-selection filters) gives a concrete blueprint for separating concerns in a production feed system. The parallel candidate source design, merging in-network and out-of-network content, is the main reference for how X structures content discovery at scale.

Skip if:

Your feed serves under a few hundred thousand users where a heuristic ranker would meet your needs. The X Algorithm's architecture addresses constraints that appear at larger scale; a simpler system may serve smaller platforms adequately.

Trust and safety engineers studying production pipelines

The labeling path covers the full pipeline from content classifiers (Grox, CLIP, media-model-proxy) through account scoring models (Agatha, BDSM, user-cred-v2) to enforcement (Scarecrow, Botmaker, abuse-enforcement-service). The appeal-protection logic in abuse-ledger-service shows how to prevent enforcement from re-applying an overturned action.

Skip if:

You are building a moderation system for content types other than short-form social posts. The labeling architecture is specific to X's post format and engagement model.

Researchers studying algorithmic transparency

The codebase reveals the scoring weights, filter conditions, and labeling rules used in X's For You feed. The August 2026 updates added comments clarifying how weights work to reduce misinterpretation. The Under the Hood transparency tool code shows how the system surfaces its own enforcement decisions to users.

Skip if:

You need a citable academic benchmark or dataset with reproducible evaluation metrics. X Algorithm is production source code, not a research artifact with standard evaluation protocols.

the problem

The problem it solves#

Building a content ranking system for a social or content platform is not a single engineering decision. It requires a retrieval layer that merges posts from accounts a user follows with posts from accounts they do not follow, a ranking model trained on engagement signals, visibility filtering for spam and policy compliance, and a continuous labeling pipeline that classifies accounts and content asynchronously.

Commercial recommendation APIs like BytePlus Recommend and Shaped handle this complexity as a managed service, but they treat the ranking logic as a black box. Teams that need to audit why content surfaces or suppress specific signals have no path to inspect or modify the model. For platforms in regulated industries or with specific editorial policies, an opaque third-party recommendation engine creates compliance and trust problems that a managed API cannot resolve.

how X Algorithm solves it

How it solves it#

Phoenix transformer ranking model

The Phoenix model predicts per-post action probabilities (like, share, dwell time, report) and combines them into a single score using weights defined in the code. The ranking code, model training code, and synthetic data generation are all included in the repository, so a proof-of-concept training run is possible without production data.

Multi-source candidate retrieval

Thunder serves in-network posts from followed accounts, held in memory for low-latency access. Phoenix retrieval finds out-of-network posts using a two-tower model. SimClusters adds a third source using cluster similarity. All three run in parallel and feed into the same ranking pipeline, letting the system blend familiar and discovered content.

Visibility filtering pipeline

The visibility-filtering service decides per-post-per-viewer whether to show a post, drop it, or show it behind an interstitial. It reads labels produced by Scarecrow (event-driven rule engine), Botmaker (rule definitions), and account-scoring models. This runs after ranking, so visibility decisions are separate from order decisions.

Labeling and abuse enforcement

Scarecrow processes events in real time using rules from Botmaker-rules, applying labels to posts and accounts. Abuse-enforcement-service reads model scores and can label, challenge, or suspend accounts. Abuse-ledger-service tracks appeals, preventing enforcement from re-applying an action that was overturned on appeal.

Configurable pipeline stages

Each stage in the request path can be switched on and off individually using configuration parameters in home-mixer/params/param.rs. This makes it possible to study individual components in isolation, run the pipeline with selected stages disabled, or experiment with alternative implementations of a specific step.

strengths · trade-offs

Strengths and trade-offs#

Strengths

Full production architecture in one repositoryX's For You feed serves hundreds of millions of users. The codebase covers every layer: retrieval, ranking, filtering, labeling, and enforcement. ML teams building recommendation systems typically reference papers or toy implementations; this is a working production system with ranking weights, training code, and the complete request pipeline included.
Apache-2.0 license permits commercial adaptationApache-2.0 permits commercial use, modification, and distribution with no copyleft requirements. You can adapt the ranking model, retrieval components, or labeling pipeline for a proprietary product without triggering license obligations on your own codebase, provided you include the Apache-2.0 notice and state any modifications.
Active development tracking live production changesThe repository received substantive updates throughout 2026: retrieval trained to surface posts people read all the way through (October 2026), improved dwell-time estimation in the ranking model (October 2026), and the addition and removal of the Brazil 2026 election filter. The last push was 2026-10-31. The codebase tracks live production changes, not a static snapshot.
Under the Hood transparency layerThe September 2026 update added a transparency tool letting users see aggregate label statistics that limit their content visibility, including whether posts were withheld following a legal demand and in which country. This transparency layer is part of the open-sourced code, showing how the system surfaces its own enforcement decisions.

Trade-offs

-Reference architecture, not a deployable serviceThe repository ships source code and model training files, not binary releases or Docker images. Running any component requires building Rust services, provisioning ML training infrastructure, and assembling the labeling pipeline. There is no step-by-step deployment guide, so teams will invest significant engineering time before any part of the system produces results.
-Rust expertise required throughoutThe codebase is written in Rust. Teams without Rust proficiency will face a significant learning curve before they can read, modify, or build from the source. ML model training for Phoenix adds complexity on top of the Rust infrastructure. Neither the ranking model nor the retrieval components can be adapted without understanding the Rust codebase.
-53 open issues with internal-team ownershipThe repository has 53 open issues as of 2026-11-01. X's internal team owns the codebase and drives updates based on production needs. Teams adapting components for external use should plan to resolve issues and gaps independently, rather than expecting a community-driven fix cycle or external contributor base.

versus alternatives

X Algorithm vs alternatives#

BytePlus Recommend is a managed recommendation API from ByteDance. X Algorithm is the open-sourced source code of X's For You feed ranking system, written in Rust under Apache-2.0. The core difference is deployment model: one is a managed API, the other is source code you build and host yourself.

Feature	X Algorithm	BytePlus Recommend
License	Apache-2.0	Proprietary
Self-hosting	Required (source only)	No (managed API)
Infrastructure	Self-managed	Vendor-managed
Ranking transparency	Full source code	Black-box API
Primary language	Rust	N/A (API service)

X Algorithm is the better choice when your team needs full transparency into ranking logic, must audit or modify scoring weights, or cannot depend on a third-party API for compliance reasons. BytePlus Recommend is the better fit when you need a managed recommendation API that handles serving and model updates without infrastructure investment.

X Algorithm vs Shaped

Shaped is a managed recommendation API service. Like BytePlus Recommend, it abstracts infrastructure and model training into a hosted service. X Algorithm differs by exposing every layer of the pipeline as Rust source code.

Feature	X Algorithm	Shaped
License	Apache-2.0	Proprietary
Self-hosting	Required (source only)	No (managed API)
Deployment	DIY infrastructure	Fully managed
Ranking transparency	Complete source	Black-box API

Shaped suits teams that need a recommendation API running quickly, without a dedicated ML platform team. X Algorithm suits teams for whom transparency, control, and independence from vendor pricing are worth the infrastructure engineering cost.

install · quick start

Quick start#

bash

Clone the repository for local setup and code exploration.
```bash
git clone https://github.com/xai-org/x-algorithm.git
```

tech stack · detected from GitHub

What it's built on#

Languages: C++JavaPythonRustScala
Frameworks: FastAPI
Infrastructure: AWS
Messaging: Kafka
Cache: Redis

frequently asked

FAQ#

Can I run X Algorithm as a recommendation service?

Not directly from a single install. The repository contains source code and model training files, not a containerized service. Running any component requires building Rust services, provisioning ML training infrastructure, and wiring up the labeling pipeline. You can adapt specific components, such as the Phoenix ranking model or SimClusters retrieval, for your own system, but there is no documented path from clone to running service.

Is X Algorithm licensed for commercial use?

Yes. It is licensed under Apache-2.0, which permits commercial use, modification, and distribution. You can adapt and use the code in a proprietary product without triggering copyleft obligations on your own codebase, provided you include the Apache-2.0 license notice and state any modifications you made.

What programming language does X Algorithm use?

The primary language is Rust. The Phoenix ML training code adds model training workflows on top of the Rust services. Reading and adapting the codebase requires Rust proficiency; the system is not accessible to teams working only in Python or TypeScript without significant translation work.

How does the Phoenix ranking model score posts?

Phoenix predicts per-post probabilities of each action a viewer might take (like, share, dwell time, report) and combines them into a single score using weights defined in the code. As the August 2026 update clarified, those weights scale predicted probabilities, not raw engagement counts. Scoring also applies repeated-author decay, an out-of-network discount, and a new-author boost before final ranking.

How does X Algorithm differ from managed recommendation APIs like BytePlus Recommend or Shaped?

Managed APIs handle model training, serving, and infrastructure for you. X Algorithm provides source code but requires you to build and operate all of that yourself. The trade-off is complete control and transparency over every ranking decision versus the engineering cost of running the full pipeline. Managed APIs suit teams that need a working recommendation system quickly; X Algorithm suits teams for whom transparency and vendor independence are worth the infrastructure investment.

also worth a look

Similar open-source tools#

FckSignups

InCustomer Support

Open-source tools that work instantly, no signup required

4,531 GitHub stars
Licence GPL-3.0

Unlimited-OCR

Alternative toGoogle Document AI

Document OCR with unlimited page depth, MIT licensed, self-hosted.

30,123 GitHub stars
Licence MIT

Vllm

Alternative toTogether AI

High-throughput LLM inference server with OpenAI-compatible API

95,878 GitHub stars
Licence Apache-2.0

Tokenspeed

Alternative toFireworks AI

MIT-licensed LLM inference engine built for agentic coding workloads.

2,832 GitHub stars
Licence MIT

Marker

Alternative toUnstructured.io

Convert documents to clean markdown and JSON with open source AI.

40,704 GitHub stars
Licence Apache-2.0

Chat2DB

Alternative toDataGrip

AI text-to-SQL client for 40+ databases; local-first, self-hosted

28,348 GitHub stars
Licence Apache-2.0 with additional commercial restrictions (LicenseRef-Chat2DB)

by category

Open source tools in these categories#

446

12,855 GitHub stars
Licence AGPL-3.0
#Email

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

6,242 GitHub stars
Licence Apache-2.0
#AI Agents

SnitchFeed

Live monitoring on LinkedIn, X, Reddit & more. Agentic social listening for startups.

Marketing & Growth

Advertise your product

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

Feature	X Algorithm	BytePlus Recommend
License	Apache-2.0	Proprietary
Self-hosting	Required (source only)	No (managed API)
Infrastructure	Self-managed	Vendor-managed
Ranking transparency	Full source code	Black-box API
Primary language	Rust	N/A (API service)

X Algorithm vs Shaped

Feature	X Algorithm	Shaped
License	Apache-2.0	Proprietary
Self-hosting	Required (source only)	No (managed API)
Deployment	DIY infrastructure	Fully managed
Ranking transparency	Complete source	Black-box API

Who X Algorithm is for#

ML engineers building content ranking systems

Skip if:

Platform engineers designing feed architecture

Skip if:

Trust and safety engineers studying production pipelines

Skip if:

Researchers studying algorithmic transparency

Skip if:

The problem it solves#

How it solves it#

Phoenix transformer ranking model

Multi-source candidate retrieval

Visibility filtering pipeline

Labeling and abuse enforcement

Configurable pipeline stages

Strengths and trade-offs#

Strengths

Trade-offs

X Algorithm vs alternatives#

X Algorithm vs BytePlus Recommend

X Algorithm vs Shaped

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Data & Analytics

Backend Development

Find open source alternatives to paid software

Who X Algorithm is for#

ML engineers building content ranking systems

Skip if:

Platform engineers designing feed architecture

Skip if:

Trust and safety engineers studying production pipelines

Skip if:

Researchers studying algorithmic transparency

Skip if:

The problem it solves#

How it solves it#

Phoenix transformer ranking model

Multi-source candidate retrieval

Visibility filtering pipeline

Labeling and abuse enforcement

Configurable pipeline stages

Strengths and trade-offs#

Strengths

Trade-offs

X Algorithm vs alternatives#

X Algorithm vs BytePlus Recommend

X Algorithm vs Shaped

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Data & Analytics

Backend Development

Find open source alternatives to paid software