JwT — samples

Audio samples from lgestin/JwT, a flow-matching TTS over raw-audio patches with rolling-window denoising.

Single-speaker model trained on LJSpeech; prompts below are out-of-domain text. Sample rate: 11.025 kHz. Model: 57M-parameter DiT (12 layers, dim 512, 8 heads), trained for ~475k steps on a single A100.

Proof of concept

This run was deliberately constrained — single-speaker LJSpeech, 11.025 kHz, 57M parameters, single A100 — to validate the core hypothesis (rolling flow matching on raw waveform patches trains stably and converges to intelligible speech) without scale confounds. Now that it works, the agenda is scaling in every direction.

Phonetic diagnostics

Pangrams and Harvard sentences — comparable across TTS systems.

01	The quick brown fox jumps over the lazy dog.	ðə kwˈɪk bɹˈWn fˈɑks ʤˈʌmps ˈOvəɹ ðə lˈAzi dˈɔɡ.
02	She sells seashells by the seashore.	ʃˌi sˈɛlz sˈiʃˌɛlz bI ðə sˈiʃˌɔɹ.
03	Peter Piper picked a peck of pickled peppers.	pˈiTəɹ pˈIpəɹ pˈɪkt ɐ pˈɛk ʌv pˈɪkəld pˈɛpəɹz.
04	The birch canoe slid on the smooth planks.	ðə bˈɜɹʧ kənˈu slˈɪd ˌɔn ðə smˈuð plˈæŋks.

Famous lines

Literary quotes, in-distribution prose.

05	It was the best of times, it was the worst of times.	ˌɪt wʌz ðə bˈɛst ʌv tˈImz, ɪt wʌz ðə wˈɜɹst ʌv tˈImz.
06	To be, or not to be, that is the question.	tə bˈi, ɔɹ nˌɑt tə bˈi, ðˈæt ɪz ðə kwˈɛsʧᵊn.
07	All happy families are alike; each unhappy family is unhappy in its own way.	ˈɔl hˈæpi fˈæmᵊliz ɑɹ əlˈIk; ˈiʧ ˌʌnhˈæpi fˈæmᵊli ɪz ˌʌnhˈæpi ɪn ɪts ˈOn wˈA.

Prosody

Questions, parallel structure, parentheticals.

08	How much wood would a woodchuck chuck if a woodchuck could chuck wood?	hˌW mˈʌʧ wˈʊd wʊd ɐ wˈʊdʧˌʌk ʧˈʌk ɪf ɐ wˈʊdʧˌʌk kʊd ʧˈʌk wˈʊd?
09	We came, we saw, we conquered.	wˌi kˈAm, wi sˈɔ, wi kˈɑŋkəɹd.
10	Yes, but, as I was saying, the matter remains unresolved.	jˈɛs, bˌʌt, æz ˌI wʌz sˈAɪŋ, ðə mˈæTəɹ ɹəmˈAnz ˌʌnɹəzˈɑlvd.

Long passages

Streaming holds up over time.

11	In a hole in the ground there lived a hobbit. Not a nasty, dirty, wet hole, filled with the ends of worms and an oozy smell.	ɪn ɐ hˈOl ɪn ðə ɡɹˈWnd ðɛɹ lˈɪvd ɐ hˈɑbət. nˌɑt ɐ nˈæsti, dˈɜɹTi, wˈɛt hˈOl, fˈɪld wɪð ði ˈɛndz ʌv wˈɜɹmz ænd ɐn ˈuzi smˈɛl.
12	The Atlantic Ocean is the second largest of the world's five oceans, covering roughly one-fifth of the Earth's surface.	ði ətlˈæntɪk ˈOʃən ɪz ðə sˈɛkənd lˈɑɹʤᵻst ʌv ðə wˈɜɹldz fˈIv ˈOʃənz, kˈʌvəɹɪŋ ɹˈʌfli wˌʌnfˈɪfθ ʌv ði ˈɜɹθs sˈɜɹfəs.
13	He had been told by many that the journey would be difficult, but nothing had prepared him for the silence of the forest at dawn.	hˌi hæd bɪn tˈOld bI mˈɛni ðæt ðə ʤˈɜɹni wʊd bi dˈɪfəkəlt, bˌʌt nˈʌθɪŋ hæd pɹipˈɛɹd hˌɪm fɔɹ ðə sˈIləns ʌv ðə fˈɔɹəst æt dˈɔn.

Self-referential

The model describing the system that generated it.

14	This audio was generated frame by frame, with no codec and no vocoder.	ðˌɪs ˈɔdiO wʌz ʤˈɛnəɹˌATᵻd fɹˈAm bI fɹˈAm, wɪð nˈO kˈOdˌɛk ænd nˈO vˈOkˌOdəɹ.
15	Flow matching learns to transport noise toward data along a continuous trajectory.	flˈO mˈæʧɪŋ lˈɜɹnz tə tɹænspˈɔɹt nˈYz tˈɔɹd dˈATə əlˈɔŋ ɐ kəntˈɪnjəwəs tɹəʤˈɛktəɹi.
16	The model speaking these words predicts raw waveform samples, one patch at a time.	ðə mˈɑdᵊl spˈikɪŋ ðiz wˈɜɹdz pɹidˈɪkts ɹˈɔ wˈAvfˌɔɹm sˈæmpᵊlz, wˈʌn pˈæʧ æt ɐ tˈIm.

⚠️ Known failure modes

EOS detection and attention alignment are the main fragility in the current model. Each clip below uses the same prompt as a working sample above — same input, different outcome — and motivates EOS robustness as the top capability priority in Next steps.

F1	How much wood would a woodchuck chuck if a woodchuck could chuck wood? Attention misalignment — phonemes drop or repeat mid-utterance under heavy lexical repetition.
F2	The quick brown fox jumps over the lazy dog. EOS false positive — stop detector fires early and truncates the sentence.
F3	He had been told by many that the journey would be difficult, but nothing had prepared him for the silence of the forest at dawn. EOS false negative — stop detector fails to fire, generation runs on past sentence end.

Next steps

📈 Scale

Bigger transformer. The current 12L / dim-512 was sized for one-GPU iteration speed; nothing about the architecture suggests it won't keep paying off.
Higher sample rate — 11.025 kHz → 22.050 kHz → 44.1 kHz.
Patch size × dimension × sample-rate sweep — map the tradeoff space as model and data grow.
Multi-speaker training — move from LJSpeech to a multi-speaker corpus and condition generation with a speaker utterance.

🧠 Capabilities

EOS robustness — RMS-threshold detector and attention-map stopping can sometimes fail and cut generation short or let it run too long. Top priority.
Semantic text embedding to replace the phoneme tokenizer and widen the conditioning signal.
Classifier-free guidance (CFG).
Perceptual / adversarial losses — the JWT architecture supports any signal-space loss as drop-in (mel L1 is the only one currently used); explore perceptual losses à la PixelGen and lightweight adversarial supervision.

⚡ Inference performance

Shrink the rolling window via short finetune — lower per-frame latency without retraining from scratch.
KV caching for generated frames and text.