The complete arXiv corpus as a Hugging Face dataset

Melvin Vivas · X post · 2026-09-20 · Open on X

Topics: AI Agents, Tool Use & MCP, Retrieval-Augmented Generation (RAG) · Level: intermediate

Summary

Shares a Hugging Face dataset holding the whole arXiv site: 3,148,796 papers in every version, as LaTeX, PDF, PostScript and HTML, about 16 TB in total. The creator suggests giving it to your agent as a research source.

Key points

• Dataset: arxiv-complete on Hugging Face (user secemp9).
• Contains 3,148,796 papers, including every version of each.
• Formats: LaTeX source, PDF, PostScript and HTML.
• Total size is about 16 TB, so plan to stream or download only parts of it.
• Suggested use: give it to an AI agent as a research or knowledge source.

Resources mentioned

• arxiv-complete (Hugging Face dataset) · dataset · huggingface.co · free
  The full arXiv archive (3.1M papers, every version, in LaTeX, PDF, PS and HTML, 16 TB) hosted on Hugging Face.
• arXiv · website · arxiv.org · free · recommended by both Bashiri Smith & Melvin Vivas
  A free archive of research preprints, with daily listings of new papers in fields like Artificial Intelligence, Machine Learning, and Computational Engineering, Finance, and Science.
  Also in: 7 Habits to Become an AI Engineer: Books, Tooling, Research & Shipping (Bashiri Smith on Facebook · notes), Generate Job-Worthy AI Project Ideas from Recent Research Papers + ChatGPT (Bashiri Smith on Facebook · notes), Software Engineer to AI Engineer: Job Boards, Stack, Projects and Learning Sites (Bashiri Smith on Facebook · notes), Red Flags That Hurt Your Chances of Landing an AI-Era Coding Job (Bashiri Smith on Facebook · notes)

Try this

• Explore the arxiv-complete dataset on Hugging Face and connect it to your agent as a source.
• Build a research agent or RAG system on top of a subset of the arXiv dataset (for example, the LaTeX or HTML of one field).