While last time I setup the code to enact my epic transfer learning plan, this week was about actually putting some data in. Thankfully, I was able to find a Dr. Seuss corpus as well as a set of Aesop’s Fables. I thought it would make sense to train on Dr Seuss first, being nominally simpler language, and then transfer up to more complex models.

During the stream, there’s not enough time to train these models to completion. But offline, I let the biggest model train overnight on Dr Seuss. The result? Well, it was something:

oh theer called a call. there they call all we likes. i hall now house do not hall. what a lot og they is nod, and the good! they should then a shall. i was for they are dear. and then this well feet of a fish. i like thing ase ase things and sis. you call tat a say. not is a good! and the good gust a lido! and then he this one hood. herd! i have a down. and when we said, oh u pat he to os of the, he head come old. i have no shad was to then moush box of therr will sing. whele theyee nerettleeee tleetle bedttle butp as a they… homle ha heold. then he say! i wish not his, and she call a pat feat the there. here. then thing house like. whele, thele things whes to! hees they thisgs they gave howtele seeetletle beetles. but, what theve lae fleas. whis is she theid head a shoull. thenee stord a chat the good fish, said thing sand and would bat bere

With the Bardic Lore model spitting out Shakespearean sounding (if nonsensical) text, I wanted to step back and try an iterative training process. I learned to read as a kid reading well, kid’s books. They have simple language, so it should be easier for a machine to learn as well. And ideally, we can use a simple model from the simple data to start training a more complex model on more complex data (like chapter books). While I didn’t have time to find a suitable data set, I did implement the syntax of this series of transfer learning. It’s like advancing through grade school for our machine learning reader.

If you have a good data set of kid’s books in a text format, I’d love to give this model a whirl. I had wanted to use Dr Seuss (and so get out some amusing phrases from the model), but that’s not in the public domain. Maybe children’s poems or fairy tales would be suitable. Hmm, or perhaps an open-source work written by a human in the style of Dr Seuss would be good enough to get a model going. All ideas welcome!

After the Keras 2 Launch Video, it’s time to get back new models, which means more regenerating Shakespeare. Last time, we stripped down all the Bard’s works to a small set of characters and setup a basic LSTM model. But after training, it spit out mainly white space. Not very impressive. Since then, I realized that I significant number of lines in the text are leading whitespace with a single digit (as for the sonnet numbers). These will no doubt pollute the model, especially if the number of white spaces was greater than the unroll length. Removing extraneous spaces, the model improves.

Training the model briefly during the stream, we saw it at least learn how long words are. Offline, I let it learn overnight, reaching about 50% validation accuracy. Code is available on GitHub. Nothing too amazing, but let’s take a look at some generated text:

r els entreat me to learn me hath the death to make with a man to can the head; englon came our duke of gentle wood of my house with thy sends of lading and dumpers of the stanf than so much that the world of hot stand a prince; but i am to do stand, and i have the enemy of the life and thou doth but you do the stand of the fortunes of soul to conferer. my lord, and the garter are or to the pardon of the graces as so, and he would not the man, when they are good nor and your or others personal use only, and are but the while of my gracious bottom of the true man, i will you shall for the son of the horses. but i shall see him be a fair duke of the which i should to arm’d it of the deadful honest of the are may i will did the stans bowes a sort the rage as the prince, that i will see the capalous lords. but in the stand the complete bear a world with like the sent of the morting sail. who should come to my lords of the prince to deserve the stay of the lady but the more heart

Depending on your opinion of Middle English, you might say this is just as intellegible as the source text :p

As I’ve enjoyed Keras for a long time, I was happy to see the launch of Keras 2.0. From a user perspective most of the changes are cosmetic; you still specify layers a line at a time and compile your model using a backend of either Theano or TensorFlow. The one hitch that caught me was that Keras 2.0 needs Theano 0.9 or higher, which hasn’t been released yet. And Theano changed something in their backend about how they internally store convolutional layers. Short story: some of my visualizations broke.

Overall, I still enjoy the library. As I incidentally also upgraded SciPy, I got to realize some speed ups there. I’m excited to hear that Keras 2 is more of an API specification, so supporting more backends and non-Python frontends will be easier in the future. Most likely, though, I’ll keep on using it in Python with Theano for practical machine learning. It makes the implementation easy, letting me focus on the fun part of asking the right question in machine learning.

This video kicks off a new series, “Bardic Lore”. You may have heard of the power of Recurrent Neural Networks to generate human-like text. While this takes quite a bit of training, I wanted to implement a modest version of this to learn and demonstrate the details. There’s various existing implementations emulating different authors, but I grok it best when I build it myself.

So far, we got the data preparation started and worked through the simplest possible example. The data I’m using is the same Shakespeare corpus I used to model fonts, only now instead of turning it into pictures, we’re actually using the text itself. To simplify the data, I lowercased everything and removed a handful of exraneous punctuation; I assure you the Bard is kept intact using just 34 unique characters. The template I followed to get started is conveniently enough in the Keras documentation. This is only a starter, but it shows how to do the simple thing: convert all your data to 1-hot format and shove it into a model. Adding more layers or tweaking the type shouldn’t alter our framework too much.

I have trained the model briefing; it’s about 30 min/epoch, so it takes awhile. But it’s still stuck on outputting spaces for the moment. Once I have something vaguely respectable (not great but not total junk), I’ll post a github repo.