Skip to content
Hanzo AI
Try Hanzo
Language: English (United States)

Research on learningwithout labels

Jinhanzoai/jin

Read the docsView on GitHub

Abstract

Eyes jitter constantly and the brain predicts what the jitter will do to the scene, so the model shifts an image, predicts how its embedding moves, and shifts back to check it lands where it started.

a input

b embedding

c losses

Huber
cycle-consistency
VICReg

Figure 1. Saccade JEPA, step by step: image at rest

What is in the repository

I-JEPA

A clean implementation of the image variant: a context encoder, a target encoder that is its moving average, and a predictor that has to guess the target embeddings. Both a ViT and an energy-transformer backbone.

Figure 2
  • acontext encoder
  • btarget encoder, its moving average
  • cpredictor
  • dtarget embeddings

SaccadeJEPA

Ours. Eyes jitter constantly and the brain predicts what the jitter will do to the scene, so the model shifts an image, predicts how its embedding moves, and shifts back to check it lands where it started.

Figure 3
  • ashift the image
  • bpredict how its embedding moves
  • cshift back
  • dlands where it started

Three lossesat once

Huber loss on the prediction, a cycle-consistency term that punishes drift on the round trip, and the invariance and covariance parts of VICReg to stop the representation collapsing.

Figure 4
  • aHuber loss on the prediction
  • bcycle-consistency: drift on the round trip
  • cVICReg: invariance and covariance

Maskedautoencoding

An MAE with self-distillation sits alongside, for the comparison that matters: is a decoder worth its parameters when the goal is a good representation rather than a good reconstruction?

Figure 5
  • athe image, masked
  • brepresentation
  • cdecoder
  • dreconstruction

Ways to seewhat it learned

Self-supervised training has no validation loss to watch, so there are linear probes, KNN, attention-map visualisation in a Dash dashboard, correlation dimension, and UMAP coloured by true class.

Figure 6
  • aUMAP coloured by true class
  • battention-map visualisation

The papers

Grant proposals on hierarchical JEPA for edge AI, where the interesting constraint is learning useful representations on hardware that cannot phone home.

Figure 7
  • ahierarchical JEPA
  • bedge AI
  • chardware that cannot phone home

Make the saccade yourself

Figure 8

Drag the image. The predictor guesses where its embedding goes; let go, and the image shifts back and has to land where it started.

drag the image

a input

b embedding

c losses

  • Huber
  • cycle-consistency
  • VICReg

hanzoai/jinLicense: Apache-2.0

Read the code

Read the docs