#pretraining

10 curated events

papersTODAY 04:00 UTC

Knowledge-enhanced approach proposed for single-cell foundation models

A new arXiv paper examines how single-cell foundation models depend on large transcriptomic pretraining datasets, noting that adding more data brings diminishing returns at rising computational cost. The authors' data scaling analysis suggests incorporating structured biological knowledge could improve efficiency instead of relying on scale alone. The work points toward knowledge-enhanced pretraining as an alternative direction for the field.

foundation-models/single-cell/pretraining/research

papersTODAY 04:00 UTC

Multi-View Molecular Pretraining Combines Hierarchical Graphs With Contextualized Fingerprints

A new arXiv paper proposes a molecular representation learning approach that combines multiple views rather than relying on a single one. It pairs hierarchical graph modeling of atom-bond topology with contextualized molecular fingerprints to improve property prediction. The goal is representations that generalize from limited labeled data to structurally novel compounds.

research/molecular-ml/graph-neural-networks/pretraining

papersTODAY 04:00 UTC

MoARa Technique Targets Faster Low-Rank LLM Pre-training

A new arXiv paper introduces MoARa, a method for low-rank gradient projection aimed at cutting the memory used by optimizer states during large language model pre-training. The authors argue that existing approaches still need too many steps and too much wall-clock time to reach a given quality level, and trace this to two design choices they address with module-aware rank allocation and structure-preserving decomposition. The work is a research preprint and has not yet been peer reviewed.

llm/pretraining/efficiency/research

papersTODAY 04:00 UTC

Study examines domain-specific pretraining effects on Arabic-English code-switching models

A new arXiv paper looks at how a model's pretraining domain profile affects Transformer performance on digital pragmatics in Arabic-English code-switched text. It compares MARBERT and XLM-R against a general-purpose BERT baseline. The work focuses on whether domain-targeted pretraining yields better results for this kind of mixed-language discourse.

arxiv/multilingual/code-switching/pretraining

papersTODAY 04:00 UTC

Proprioception-Anchored Cross-Modal Pretraining for Zero-Shot Sim-to-Real Assembly

The paper introduces a pretraining method that uses proprioceptive signals as an anchor to align different sensory modalities, aiming to help robots handle contact-rich assembly tasks. Such tasks demand submillimeter precision and accurate force interpretation during sustained contact, which makes sim-to-real transfer difficult. The approach targets zero-shot deployment of policies trained in simulation onto real hardware. This is a replacement listing on arXiv cs.AI.

robotics/sim-to-real/pretraining/manipulation

papersTODAY 04:00 UTC

Pretraining Approach Aims to Cut Labeled Data Needs for Brain-Computer Interface Decoders

A new arXiv paper examines pretraining methods for neural decoders used in brain-computer interfaces. Because training a high-performing decoder normally requires large labeled datasets from each new subject, the work targets ways to lower that annotation burden. The abstract is truncated in the source, so full results are not yet available.

brain-computer-interface/pretraining/sample-efficiency/neural-decoding

papersTODAY 04:00 UTC

Study Scales JugnuLM Language Models From 53M to 110M Parameters

A new arXiv paper examines how a fixed sub-150M pretraining recipe behaves when model size grows from 53.5M to 109.7M parameters. Both models use a Qwen3-style decoder with grouped-query attention, RoPE, SwiGLU, RMSNorm, QK-Norm and a z-loss, trained on FineWeb-Edu data, so only scale and depth differ. The work compares the 53M and 110M variants to isolate the effects of added capacity in this small-model regime.

llm/scaling/pretraining/research

papersTODAY 04:00 UTC

Tabby: Open Pretraining Recipe Released for Time Series Foundation Models

Researchers introduce Tabby, a long-context probabilistic foundation model designed for time series data, built on an encoder-only patch Transformer architecture. The release includes a fully open account of the pretraining process, covering the decisions and components behind the model's construction. The work aims to make time series foundation model development more reproducible and accessible.

time-series/foundation-models/open-source/pretraining

papersSEP 12 04:00 UTC

arXiv paper explores structural priors from non-language data for language learning

A new arXiv preprint examines whether pre-training on non-language data can create useful priors that make natural language learning more data- and compute-efficient. The authors frame the work as a study of structural transfer, aiming to cut the heavy resource demands of training language models. The abstract does not report specific benchmarks or results in the provided excerpt.

research/data-efficiency/pretraining/arxiv

papersSEP 11 04:00 UTC

Paper Proposes Output Embedding Centering to Curb LLM Pretraining Instability

A new arXiv preprint introduces a method called output embedding centering aimed at reducing output logit divergence, a form of training instability that tends to appear late in large language model pretraining. The authors position it against commonly used mitigations such as z-loss. The work is a research contribution rather than a released model or product.

pretraining/training-stability/llm/research