EleutherAI

/

sae-llama-3.1-8b-64x

like 9

Follow

EleutherAI 529

TL:DR; SAEs loaded do not have the W_dec field in the repr but you can extract it through sae.W_dec.

repr of pretrained SAEs:
Sae(
(encoder): Linear(in_features=4096, out_features=262144, bias=True)
)

As can be seen above, decoder is not printed. While sae.encoder corresponds to the nn.Linear for encoding, there is no sae.decoder field. Instead, you have to use the W_dec attribute.

Flagging this here in case anyone else is confused when they load the model and only see the encoder.

See screenshot below