Cached SAE/transcoder acts stored in CSR format. Not especially optimized for others' use/fleshed out. It's mostly just a space for me to place and download activations.
If you want to use them, do
import torch
from huggingface_hub import hf_hub_download
def load_feat_acts(fname, only_active_docs=False):
local_path = hf_hub_download(repo_id="noanabeshima/tiny_model_cached_acts", filename=fname)
csr_kwargs = torch.load(local_path)
# The matrices might be stored in space-efficient formats that're incompatible with torch's sparse csr tensor.
# Convert them back before constructing the matrix.
csr_kwargs['crow_indices'] = csr_kwargs['crow_indices'].int()
csr_kwargs['col_indices'] = csr_kwargs['col_indices'].int()
values = csr_kwargs['values'].float()
csr_kwargs['values'] = values/values.max()
feat_acts = torch.sparse_csr_tensor(**csr_kwargs)
return feat_acts
feat_acts = load_feat_acts(f"mlp_map_test/M2_S-2_R1_P0/{300}.pt").to_dense()
The activations are from 13% of the train split of https://huggingface.co/datasets/noanabeshima/TinyModelTokIds. I gather all the activations per feature and then take the smallest activations prefix so that each feature has at least 3K documents on which it activates and 12K token-activations.